Encoding apparatus and encoding method as well as decoding apparatus and decoding method

ABSTRACT

The encoding apparatus transmits reduction filter information that reduces tap coefficients for individual ones of a plurality of classes determined by learning that uses a student image equivalent to a first image obtained by addition of a residual of prediction encoding and a prediction image and a teacher image equivalent to an original image corresponding to the first image. The decoding apparatus accepts the reduction filter information and performs prediction arithmetic operation using tap coefficients obtained using the reduction filter information to perform a filter process for the first image to generate a second image. The present technology can be applied, for example, an encoding apparatus and a decoding apparatus of an image.

CROSS REFERENCE TO PRIOR APPLICATION

This application is a National Stage Patent Application of PCTInternational Patent Application No. PCT/JP2017/015522 (filed on Apr.18, 2017) under 35 U.S.C. § 371, which claims priority to JapanesePatent Application Nos. 2016-092651 (filed on May 2, 2016) and2017-003466 (filed on Jan. 12, 2017), which are all hereby incorporatedby reference in their entirety.

TECHNICAL FIELD

The present technology relates to an encoding apparatus and an encodingmethod as well as a decoding apparatus and a decoding method, andparticularly to an encoding apparatus and an encoding method as well asa decoding apparatus and a decoding method that make it possible, forexample, to improve the compression efficiency of an image.

BACKGROUND ART

For example, a classification adaptive process for converting a firstimage into a second image has been proposed previously. In theclassification adaptive process, a pixel that becomes a prediction tapto be used for prediction arithmetic operation for determining a pixelvalue of a corresponding pixel of a second image corresponding to anoticed pixel noticed in a first image is selected from within the firstimage, and the noticed pixel is classified to one of a plurality ofclasses in accordance with a fixed rule. Then, in the classificationadaptive process, a tap coefficient of the class of the noticed pixel isacquired from among tap coefficients to be used for the predictionarithmetic operation for each of a plurality of classes determined bylearning for minimizing statistical errors between a result of theprediction arithmetic operation in which a student image equivalent tothe first image is used and a teacher image equivalent to the secondimage. Then, a pixel value of the corresponding pixel is determined byperforming prediction arithmetic operation using the tap coefficient ofthe class of the noticed pixel and a prediction tap of the noticedpixel.

It is to be noted that, in regard to the classification adaptiveprocess, a technology that integrates tap coefficients of two or moreclasses (for example, PTL 1) and another technology that determines aseed coefficient from which a tap coefficient is determined bypredetermined arithmetic operation with a parameter (for example, PTL 2)have been proposed.

CITATION LIST Patent Literature

[PTL 1]

Japanese Patent No. 3890638

[PTL 2]

Japanese Patent No. 4670169

SUMMARY Technical Problem

Incidentally, for example, in prediction encoding of an image,improvement of the compression efficiency is requested.

The present technology has been made in view of such a situation asdescribed above and makes it possible to improve the compressionefficiency of an image.

Solution to Problem

The encoding apparatus of the present technology is an encodingapparatus including a filter processing section that includes aprediction tap selection section configured to select, from within afirst image obtained by addition of a residual of prediction encodingand a prediction image, pixels that become a prediction tap to be usedfor prediction arithmetic operation for determining a pixel value of acorresponding pixel of a second image, which corresponds to a processingtarget pixel that is a processing target in the first image and is to beused for prediction of the prediction image, a classification sectionconfigured to classify the processing target pixel to one of a pluralityof classes, a tap coefficient acquisition section configured to acquiretap coefficients of the class of the processing target pixel from amongtap coefficients obtained using reduction filter information thatreduces tap coefficients for individual ones of the plurality of classesdetermined by learning that uses a student image corresponding to thefirst image and a teacher image equivalent to an original imagecorresponding to the first image, and an arithmetic operation sectionconfigured to determine a pixel value of the corresponding pixel byperforming the prediction arithmetic operation using the tapcoefficients of the class of the processing target pixel and theprediction tap of the processing target pixel, and performs a filterprocess for the first image to generate the second image, and atransmission section configured to transmit the reduction filterinformation.

The encoding method of the present technology is an encoding methodincluding performing a filter process for a first image to generate asecond image, the performing a filter process including selecting, fromwithin the first image that is obtained by addition of a residual ofprediction encoding and a prediction image, pixels that become aprediction tap to be used for prediction arithmetic operation fordetermining a pixel value of a corresponding pixel of the second image,which corresponds to a processing target pixel that is a processingtarget in the first image and is to be used for prediction of theprediction image, classifying the processing target pixel to one of aplurality of classes, acquiring tap coefficients of the class of theprocessing target pixel from among tap coefficients obtained usingreduction filter information that reduces tap coefficients forindividual ones of the plurality of classes determined by learning thatuses a student image corresponding to the first image and a teacherimage equivalent to an original image corresponding to the first image,and determining a pixel value of the corresponding pixel by performingthe prediction arithmetic operation using the tap coefficients of theclass of the processing target pixel and the prediction tap of theprocessing target pixel, and transmitting the reduction filterinformation.

In the encoding apparatus and the encoding method of the presenttechnology, from within a first image that is obtained by addition of aresidual of prediction encoding and a prediction image, pixels thatbecome a prediction tap to be used for prediction arithmetic operationfor determining a pixel value of a corresponding pixel of the secondimage, which corresponds to a processing target pixel that is aprocessing target in the first image and is to be used for prediction ofthe prediction image, are selected, and the processing target pixel isclassified to one of a plurality of classes. Further, tap coefficientsof the class of the processing target pixel are acquired from among tapcoefficients obtained using reduction filter information that reducestap coefficients for individual ones of the plurality of classesdetermined by learning that uses a student image corresponding to thefirst image and a teacher image equivalent to an original imagecorresponding to the first image, and a pixel value of the correspondingpixel is determined by performing the prediction arithmetic operationusing the tap coefficients of the class of the processing target pixeland the prediction tap of the processing target pixel. A filter processis performed thereby for the first image, and a second image isgenerated. Further, the reduction filter information is transmitted.

The decoding apparatus of the present technology is a decoding apparatusincluding an acceptance section configured to accept reduction filterinformation that reduces tap coefficients for individual ones of aplurality of classes determined by learning that uses a student imageequivalent to a first image obtained by adding a residual of predictionencoding and a prediction image and a teacher image equivalent to anoriginal image corresponding to the first image, and a filter processingsection that includes a prediction tap selection section configured toselect, from within the first image, pixels that become a prediction tapto be used for prediction arithmetic operation for determining a pixelvalue of a corresponding pixel of a second image, which is used forprediction of the prediction image, corresponding to a processing targetpixel that is a processing target from within the first image, aclassification section configured to classify the processing targetpixel to one of the plurality of classes, a tap coefficient acquisitionsection configured to acquire a tap coefficient of the class of theprocessing target pixel from the tap coefficients obtained using thereduction filter information, and an arithmetic operation sectionconfigured to determine a pixel value of the corresponding pixel byperforming the prediction arithmetic operation using the tap coefficientof the class of the processing target pixel and the prediction tap ofthe processing target pixel, and performs a filter process for the firstimage to generate the second image.

The decoding method of the present technology is a decoding methodincluding accepting reduction filter information that reduces tapcoefficients for individual ones of a plurality of classes determined bylearning that uses a student image equivalent to a first image obtainedby adding a residual of prediction encoding and a prediction image and ateacher image equivalent to an original image corresponding to the firstimage, and performing a filter process for the first image to generate asecond image, the performing a filter process including selecting, fromwithin the first image, pixels that become a prediction tap to be usedfor prediction arithmetic operation for determining a pixel value of acorresponding pixel of a second image, which is used for prediction ofthe prediction image, corresponding to a processing target pixel that isa processing target from within the first image, classifying theprocessing target pixel to one of the plurality of classes, acquiring atap coefficient of the class of the processing target pixel from the tapcoefficients obtained using the reduction filter information, anddetermining a pixel value of the corresponding pixel by performing theprediction arithmetic operation using the tap coefficient of the classof the processing target pixel and the prediction tap of the processingtarget pixel.

In the decoding apparatus and the decoding method of the presenttechnology, reduction filter information is accepted which reduces tapcoefficients for individual ones of a plurality of classes determined bylearning that uses a student image equivalent to a first image obtainedby adding a residual of prediction encoding and a prediction image and ateacher image equivalent to an original image corresponding to the firstimage. Further, from within the first image, pixels are selected whichbecome a prediction tap to be used for prediction arithmetic operationfor determining a pixel value of a corresponding pixel of a secondimage, which is used for prediction of the prediction image,corresponding to a processing target pixel that is a processing targetfrom within the first image, and the processing target pixel isclassified to one of the plurality of classes. Then, a tap coefficientof the class of the processing target pixel is acquired from the tapcoefficients obtained using the reduction filter information, and apixel value of the corresponding pixel is determined by performing theprediction arithmetic operation using the tap coefficient of the classof the processing target pixel and the prediction tap of the processingtarget pixel. By this, a filter process for the first image isperformed, and a second image is generated.

It is to be noted that each of the encoding apparatus and the decodingapparatus may be an independent apparatus or may be an internal blockthat configures one apparatus.

Further, each of the encoding apparatus and the decoding apparatus canbe implemented by causing a computer to execute a program.

Further, the program that causes a computer to function as the encodingapparatus or the decoding apparatus can be provided by transmitting thesame through a transmission medium or by recording the same on arecording medium.

Advantageous Effect of Invention

With the present technology, the compression efficiency of an image canbe improved.

It is to be noted that the advantageous effect described herein is notnecessarily restrictive, and any advantageous effect described in thepresent disclosure may be applicable.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a view depicting a configuration example of an embodiment ofan image processing system to which the present technology is applied.

FIG. 2 is a block diagram depicting a first configuration example of animage conversion apparatus that performs a classification adaptiveprocess.

FIG. 3 is a block diagram depicting a configuration example of alearning apparatus that performs learning of a tap coefficient to bestored into a coefficient acquisition section 24.

FIG. 4 is a block diagram depicting a configuration example of alearning section 33.

FIG. 5 is a block diagram depicting a second configuration example ofthe image conversion apparatus that performs a classification adaptiveprocess.

FIG. 6 is a block diagram depicting a configuration example of thelearning apparatus that performs learning of a seed coefficient to bestored into the coefficient acquisition section 24.

FIG. 7 is a block diagram depicting a configuration example of alearning section 63.

FIG. 8 is a block diagram depicting another configuration example of alearning section 63.

FIG. 9 is a block diagram depicting a first configuration example of anencoding apparatus 11.

FIG. 10 is a block diagram depicting a configuration example of aclassification adaptive filter 111.

FIG. 11 is a view depicting an example of an update timing of a tapcoefficient to be used for the classification adaptive process by animage conversion apparatus 133.

FIG. 12 is a view illustrating an example of a reduction process of areduction apparatus 132.

FIG. 13 is a block diagram depicting a configuration example of alearning apparatus 131.

FIG. 14 is a block diagram depicting a configuration example of thereduction apparatus 132.

FIG. 15 is a block diagram depicting a configuration example of theimage conversion apparatus 133.

FIG. 16 is a block diagram depicting a configuration example of acoefficient acquisition section 151.

FIG. 17 is a flow chart illustrating an example of an encoding processof the encoding apparatus 11.

FIG. 18 is a flow chart illustrating an example of a prediction encodingprocess at step S21.

FIG. 19 is a flow chart illustrating an example of a classificationadaptive process performed at step S42.

FIG. 20 is a block diagram depicting a first configuration example of adecoding apparatus 12.

FIG. 21 is a block diagram depicting a configuration example of aclassification adaptive filter 206.

FIG. 22 is a block diagram depicting a configuration example of an imageconversion apparatus 231.

FIG. 23 is a block diagram depicting a configuration example of acoefficient acquisition section 244.

FIG. 24 is a flow chart illustrating an example of a decoding process ofthe decoding apparatus 12.

FIG. 25 is a flow chart illustrating an example of a prediction decodingprocess at step S75.

FIG. 26 is a flow chart illustrating an example of a classificationadaptive process performed at step S86.

FIG. 27 is a block diagram depicting a second configuration example ofthe encoding apparatus 11.

FIG. 28 is a block diagram depicting a configuration example of aclassification adaptive filter 311.

FIG. 29 is a view illustrating an example of a reduction process of areduction apparatus 321.

FIG. 30 is a block diagram depicting a configuration example of thereduction apparatus 321.

FIG. 31 is a block diagram depicting a configuration example of an imageconversion apparatus 322.

FIG. 32 is a block diagram depicting a configuration example of acoefficient acquisition section 341.

FIG. 33 is a flow chart illustrating an example of an encoding processof the encoding apparatus 11.

FIG. 34 is a flow chart illustrating an example of a prediction encodingprocess at step S123.

FIG. 35 is a flow chart illustrating an example of a classificationadaptive process performed at step S142.

FIG. 36 is a block diagram depicting a second configuration example ofthe decoding apparatus 12.

FIG. 37 is a block diagram depicting a configuration example of aclassification adaptive filter 411.

FIG. 38 is a block diagram depicting a configuration example of an imageconversion apparatus 431.

FIG. 39 is a block diagram depicting a configuration example of acoefficient acquisition section 441.

FIG. 40 is a flow chart illustrating an example of a decoding process ofthe decoding apparatus 12.

FIG. 41 is a flow chart illustrating an example of a prediction decodingprocess at step S175.

FIG. 42 is a flow chart illustrating an example of a classificationadaptive process performed at step S186.

FIG. 43 is a block diagram depicting a third configuration example ofthe encoding apparatus 11.

FIG. 44 is a block diagram depicting a configuration example of aclassification adaptive filter 511.

FIG. 45 is a block diagram depicting a configuration example of alearning apparatus 531.

FIG. 46 is a block diagram depicting a configuration example of alearning section 543.

FIG. 47 is a view illustrating a relationship between a parameter z anda tap coefficient.

FIG. 48 is a view illustrating an example of a relationship between adistribution of coefficient points and an order of a seed coefficientthat defines a relational curve that fits with the distribution ofcoefficient points.

FIG. 49 is a view illustrating a different example of a relationshipbetween a distribution of coefficient points and an order of a seedcoefficient that defines a relational curve that fits with thedistribution of coefficient points.

FIG. 50 is a block diagram depicting a configuration example of an imageconversion apparatus 532.

FIG. 51 is a block diagram depicting a configuration example of acoefficient acquisition section 562.

FIG. 52 is a flow chart illustrating an example of an encoding processof the encoding apparatus 11.

FIG. 53 is a flow chart illustrating an example of a prediction encodingprocess at step S220.

FIG. 54 is a flow chart illustrating an example of a classificationadaptive process performed at step S242.

FIG. 55 is a block diagram depicting a third configuration example ofthe decoding apparatus 12.

FIG. 56 is a block diagram depicting a configuration example of aclassification adaptive filter 611.

FIG. 57 is a block diagram depicting a configuration example of an imageconversion apparatus 631.

FIG. 58 is a block diagram depicting a configuration example of acoefficient acquisition section 641.

FIG. 59 is a flow chart illustrating an example of a decoding process ofthe decoding apparatus 12.

FIG. 60 is a flow chart illustrating an example of a prediction decodingprocess at step S275.

FIG. 61 is a flow chart illustrating an example of a classificationadaptive process performed at step S286.

FIG. 62 is a view illustrating a different example of reduction filterinformation that reduces tap coefficients for individual classesobtained by tap coefficient learning.

FIG. 63 is a view illustrating an example of reduction of tapcoefficients by a class evaluation value utilization method.

FIG. 64 is a block diagram depicting a fourth configuration example ofthe encoding apparatus 11.

FIG. 65 is a block diagram depicting a configuration example of aclassification adaptive filter 911.

FIG. 66 is a block diagram depicting a configuration example of alearning apparatus 931.

FIG. 67 is a block diagram depicting a configuration example of a tapselection section 942 and a classification section 943.

FIG. 68 is a view depicting an example of an image characteristic amountthat becomes pixel-related information.

FIG. 69 is a view depicting an example of a method by which a classconfiguration section 954 configures an initial class from first to Hthsub classes.

FIG. 70 is a view depicting a first example of a combination of aplurality of kinds of pixel-related information to be used forclassification of a noticed pixel.

FIG. 71 is a view depicting a second example of a combination of aplurality of kinds of pixel-related information to be used forclassification of a noticed pixel.

FIG. 72 is a view depicting a third example of a combination of aplurality of kinds of pixel-related information to be used forclassification of a noticed pixel.

FIG. 73 is a block diagram depicting a configuration example of adeletion apparatus 932.

FIG. 74 is a block diagram depicting a configuration example of an imageconversion section 981.

FIG. 75 is a block diagram depicting a configuration example of an imageconversion section 991.

FIG. 76 is a block diagram depicting a configuration example of acoefficient acquisition section 724.

FIG. 77 is a block diagram depicting a configuration example of a classdegeneration section 973.

FIG. 78 is a view illustrating an example of degeneration of tapcoefficients for individual initial classes by a seed coefficientutilization method.

FIG. 79 is a block diagram depicting a configuration example of alearning section 742 _(v).

FIG. 80 is a block diagram depicting a configuration example of an imageconversion apparatus 933.

FIG. 81 is a block diagram depicting a configuration example of acoefficient acquisition section 774.

FIG. 82 is a flow chart illustrating an example of the encoding processof the encoding apparatus 11.

FIG. 83 is a flow chart illustrating an example of a prediction encodingprocess at step S320.

FIG. 84 is a flow chart illustrating an example of a classificationadaptive process performed at step S342.

FIG. 85 is a block diagram depicting a fourth configuration example ofthe decoding apparatus 12.

FIG. 86 is a block diagram depicting a configuration example of aclassification adaptive filter 811.

FIG. 87 is a block diagram depicting a configuration example of an imageconversion apparatus 831.

FIG. 88 is a block diagram depicting a configuration example of acoefficient acquisition section 844.

FIG. 89 is a flow chart illustrating an example of the decoding processof the decoding apparatus 12.

FIG. 90 is a flow chart illustrating an example of a prediction decodingprocess at step S375.

FIG. 91 is a flow chart illustrating an example of a classificationadaptive process performed at step S386.

FIG. 92 is a view depicting an example of a multi-view image encodingmethod.

FIG. 93 is a view depicting a principal configuration example of amulti-view image encoding apparatus to which the present technology isapplied.

FIG. 94 is a view depicting a principal configuration example of amulti-view image decoding apparatus to which the present technology isapplied.

FIG. 95 is a view depicting an example of a hierarchical image encodingmethod.

FIG. 96 is a view depicting a principal configuration example of ahierarchical image encoding apparatus to which the present technology isapplied.

FIG. 97 is a view depicting a principal configuration example of ahierarchical image decoding apparatus to which the present technology isapplied.

FIG. 98 is a block diagram depicting a principal configuration exampleof a computer.

FIG. 99 is a block diagram depicting an example of a schematicconfiguration of a television apparatus.

FIG. 100 is a block diagram depicting an example of a schematicconfiguration of a portable telephone set.

FIG. 101 is a block diagram depicting an example of a schematicconfiguration of a recording and reproduction apparatus.

FIG. 102 is a block diagram depicting an example of a schematicconfiguration of an image pickup apparatus.

FIG. 103 is a block diagram depicting an example of a schematicconfiguration of a video set.

FIG. 104 is a block diagram depicting an example of a schematicconfiguration of a video processor.

FIG. 105 is a block diagram depicting a different example of a schematicconfiguration of the video processor.

DESCRIPTION OF EMBODIMENTS

<Image Processing System to which Present Technology is Applied>

FIG. 1 is a view depicting a configuration example of an embodiment ofan image processing system to which the present technology is applied.

Referring to FIG. 1, the image processing system includes an encodingapparatus 11 and a decoding apparatus 12.

An original image of an encoding target is supplied to the encodingapparatus 11.

The encoding apparatus 11 encodes the original image by predictionencoding such as, for example, HEVC (High Efficiency Video Coding), AVC(Advanced Video Coding), MPEG (Moving Picture Experts Group) or thelike. It is to be noted that the prediction encoding of the encodingapparatus 11 is not limited to such HEVC or the like as described above.

In prediction encoding of the encoding apparatus 11, a prediction imageof an original image is generated and a residual between the originalimage and the prediction image is encoded.

Further, in the prediction encoding of the encoding apparatus 11, an ILF(In Loop Filter) process for applying an ILF is performed for a decodingin-progress image obtained by adding the residual of the predictionencoding and the prediction image to generate a reference image to beused for prediction of the prediction image.

Here, an image obtained by performing a filter process (filtering) asthe ILF process for the decoding in-progress image is referred tosometimes as post-filter image.

The encoding apparatus 11 performs not only prediction encoding but alsolearning using a decoding in-progress image and an original image todetermine a tap coefficient or the like for performing such a filteringprocess as an ILF process that the post-filter image becomes similar tothe original image as far as possible.

Further, the encoding apparatus 11 performs a reduction process togenerate reduction filter information that reduces tap coefficients.

The ILF process of the encoding apparatus 11 is performed using tapcoefficients obtained using the reduction filter information determinedby the reduction process.

Here, learning for determining tap coefficients or the like and areduction process for generating reduction filter information can beperformed for example, for each of one or a plurality of sequences oforiginal images, for each of one or a plurality of scenes (frames eachfrom a scene change to a next scene change) of an original image, foreach of one or a plurality of frames (pictures) of an original image,for each of one or a plurality of slices of an original image, for eachof one or a plurality of lines of a block (CU, PU or the like) of a unitof encoding of a picture or for some other arbitrary unit. Further,learning for determining reduction filter information can be performed,for example, in the case where a residual obtained by predictionencoding becomes equal to or greater than a threshold value or in a likecase.

The encoding apparatus 11 transmits encoded data obtained by predictionencoding of an original image and reduction filter information obtainedby the reduction process through a transmission medium 13 or transmitsthem to a recording medium 14 so as to be recorded.

It is to be noted that generation of reduction filter information(including learning of tap coefficients as occasion demands) can beperformed by an apparatus separate from the encoding apparatus 11.

Also it is possible not only to transmit reduction filter informationseparately from encoded data and but also to place and transmitreduction filter information into and together with encoded data.

Further, learning for calculating tap coefficients or the like can beperformed not only using an original image itself (and a decodingin-progress image obtained by prediction encoding of the original image)but using an image that is different from the original image but issimilar in image characteristic amount of the original image.

The decoding apparatus 12 accepts (receives) (acquires) encoded data andreduction filter information transmitted from the encoding apparatus 11through the transmission medium 13 or the recording medium 14, anddecodes the encoded data by a method corresponding to that of theprediction encoding of the encoding apparatus 11.

In particular, the decoding apparatus 12 processes the encoded data fromthe encoding apparatus 11 to determine a residual of predictionencoding. Further, the decoding apparatus 12 adds the residual and theprediction image to determine a decoding in-progress image similar tothat obtained in the encoding apparatus 11. Then, the decoding apparatus12 performs a filter process as an ILF process using tap coefficientsand so forth obtained using the reduction filter information from theencoding apparatus 11 for the decoding in-progress image to determine apost-filter image.

In the decoding apparatus 12, the post-filter image is outputted as adecoded image of the original image and is temporarily stored as areference image to be used for prediction of a prediction image.

The filter process as an ILF process of the encoding apparatus 11 anddecoding apparatus 12 is performed by a classification adaptive process.The classification adaptive process is described below.

<Classification Adaptive Process>

FIG. 2 is a block diagram depicting a first configuration example of animage conversion apparatus that performs the classification adaptiveprocess.

Here, the classification adaptive process can be considered, forexample, as an image conversion process for converting a first imageinto a second image.

The image conversion process for converting a first image into a secondimage becomes various signal processes depending upon the definition ofthe first and second images.

In particular, for example, if the first image is an image of a lowspatial resolution and the second image is an image of a high spatialresolution, then the image conversion process can be considered as aspatial resolution creation (improvement) process for improving thespatial resolution.

On the other hand, for example, if the first image is an image of a lowS/N (Signal to Noise Ratio) and the second image is an image of a highS/N, then the image conversion process can be considered as a noiseremoving process for removing noise.

Furthermore, for example, if the first image is an image having apredetermined number of pixels (size) and the second image is an imagewhose number of pixels is made higher or lower than the number of pixelsof the first image, then the image conversion process can be consideredas a resize process for performing resizing (enlargement or reduction)of an image.

Further, for example, if the first image is a decoded image obtained bydecoding an image encoded in a unit of a block such as HEVC or the likeand the second image is an original image before encoding, then theimage conversion process can be considered as a distortion removingprocess for removing block distortion generated by encoding and decodingin a unit of a block.

It is to be noted that, in the classification adaptive process, not onlyan image but also, for example, sound can be made a target ofprocessing. The classification adaptive process whose target is soundcan be considered as an acoustic conversion process for converting firstsound (for example, sound having a low S/N or the like) into secondsound (for example, sound having a high S/N or the like).

In the classification adaptive process, a pixel value of a noticed pixelis determined by prediction arithmetic operation using tap coefficientsof a class obtained by classifying a pixel value of a noticed pixel(processing target pixel of a processing target) noticed from within thefirst image to one of a plurality of classes and prediction and pixelvalues of the number of pixels equal to that of the tap coefficients ofthe first image selected with respect to the noticed pixel.

FIG. 2 depicts a configuration example of the image conversion apparatusthat performs an image conversion process by the classification adaptiveprocess.

Referring to FIG. 2, an image conversion apparatus 20 includes tapselection sections 21 and 22, a classification section 23, a coefficientacquisition section 24 and a prediction arithmetic operation section 25.

A first image is supplied to the image conversion apparatus 20. Thefirst image supplied to the image conversion apparatus 20 is supplied tothe tap selection sections 21 and 22.

The tap selection section 21 selects pixels configuring the first imagesuccessively as a noticed pixel. Further, the tap selection section 21selects some of (pixel values of) pixels configuring the first image tobe used for prediction of (a pixel value of) a corresponding pixel of asecond image corresponding to the noticed pixel as a prediction tap.

In particular, the tap selection section 21 selects a plurality ofpixels of the first image at a spatially or temporally close positionfrom the spatio-temporal position of the noticed pixel.

The tap selection section 22 selects some of (pixel values of) pixelsconfiguring the first image to be used for classification forclassifying the noticed pixel to one of several classes as a class tap.In particular, the tap selection section 22 selects a class tapsimilarly to the selection of a prediction tap by the tap selectionsection 21.

It is to be noted that a prediction tap and a class tap may have a sametap structure or may have tap structures different from each other.

A prediction tap obtained by the tap selection section 21 is supplied tothe prediction arithmetic operation section 25, and a class tap obtainedby the tap selection section 22 is supplied to the classificationsection 23.

The classification section 23 classifies the noticed pixel in accordancewith a fixed rule, and supplies a class code corresponding to a classobtained as a result of the classification to the coefficientacquisition section 24.

In particular, the classification section 23 classifies the noticedpixel, for example, using the class tap from the tap selection section22 and supplies a class code corresponding to a class obtained as aresult of the classification to the coefficient acquisition section 24.

For example, the classification section 23 determines an imagecharacteristic amount of the notice image using the class tap. Further,the classification section 23 classifies the noticed pixel according tothe image characteristic amount of the noticed pixel and supplies aclass code corresponding to a class obtained as a result of theclassification to the coefficient acquisition section 24.

Here, as a method for performing classification, for example, ADRC(Adaptive Dynamic Range Coding) or the like can be adopted.

In the method that uses the ADRC, (pixel values of) pixels configuringthe class tap are ADRC processed, and a class of the noticed pixel isdetermined in accordance with an ADRC code (ADRC value) obtained as aresult of the ADRC process. The ADRC code represents a waveform patternas the image characteristic amount of a small region including thenoticed pixel.

It is to be noted that, in L bit ADRC, for example, a maximum value MAXand a minimum value MIN of pixel values of pixels configuring a classtap are detected, and DR=MAX−MIN is determined as a local dynamic rangeof a set and the pixel values of the pixels configuring the class tapare re-quantized to L bits on the basis of the dynamic range DR. Inparticular, the minimum value MIN is subtracted from the pixel value ofeach of the pixels of configuring the class tap and the subtractionvalues are divided (re-quantized) by DR/2^(L). Then, a bit string inwhich the pixel values of the pixels of the L bits configuring the classtap obtained as in such a manner as described above are lined up in apredetermined order is outputted as an ADRC code. Accordingly, in thecase where the class tap is processed, for example, by one-bit ADRCprocessing, the pixel values of the pixels configuring the class tap aredivided (truncate a fractional part) by an average value of the maximumvalue MAX and the minimum value MIN, and, as a result, the pixel valueof each pixel comes to be represented by 1 bit (binarized). Then, a bitstring in which the pixel values of 1 bit are lined up in apredetermined order is outputted as an ADRC code.

It is to be noted that it is possible to cause the classificationsection 23 to output, for example, a pattern of a level distribution ofthe pixel values of the pixels configuring the class tap as it is as aclass code. However, in this case, if the class tap is configured frompixel values of N pixels and A bits are allocated to the pixel value ofeach pixel, then the number of cases of the class code to be outputtedfrom the classification section 23 is (2^(N))^(A) and is a huge numberwhich increases in exponential proportion to the bit number A of thepixel values of the pixels.

Accordingly, it is preferable for the classification section 23 toperform classification by compressing the information amount of classtaps by the ADRC process described above or by vector quantization orthe like.

The coefficient acquisition section 24 stores tap coefficients forindividual classes determined by learning hereinafter described andfurther acquires tap coefficients of a class represented by a class codesupplied from the classification section 23 from among the stored tapcoefficients, namely, tap coefficient of a class of a noticed pixel.Further, the coefficient acquisition section 24 supplies the tapcoefficient of the class of the noticed pixel to the predictionarithmetic operation section 25.

Here, the tap coefficient is a coefficient equivalent to a coefficientto be multiplied by input data in a so-called tap in a digital filter.

The prediction arithmetic operation section 25 performs predeterminedprediction arithmetic operation for determining a prediction value of atrue value of a pixel value of a pixel (corresponding pixel) of a secondimage corresponding to the noticed pixel using the prediction tapoutputted from the tap selection section 21 and the tap coefficientsupplied from the coefficient acquisition section 24. Consequently, theprediction arithmetic operation section 25 determines and outputs (aprediction value of) a pixel value of the corresponding pixel, namely, apixel value of a pixel configuring the second image.

FIG. 3 is a block diagram depicting a configuration example of alearning apparatus that performs learning of a tap coefficient to bestored into the coefficient acquisition section 24.

Here, for example, it is conceived that, determining an image havinghigh picture quality (high picture quality image) as a second image anddetermining an image having low picture quality (low picture qualityimage) whose picture quality (resolution) is decreased by filtering orthe like of the high picture quality image by an LPF (Low Pass Filter),a prediction tap is selected from within the low picture quality imageand a pixel value of a pixel of the high picture quality image (highpicture quality pixel) is determined (predicted) by predeterminedprediction arithmetic operation using the prediction tap and the tapcoefficient.

For example, if linear primary prediction arithmetic operation isadopted as the predetermined prediction arithmetic operation, then thepixel value y of the high picture quality pixel is determined by thefollowing linear primary expression.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack & \; \\{y = {\sum\limits_{n = 1}^{N}{w_{n}x_{n}}}} & (1)\end{matrix}$

However, in the expression (1), X_(n) represents the pixel value of annth pixel of the low picture quality image (hereinafter referred tosuitably as low picture quality pixel) configuring the prediction tapwith respect to a high picture quality pixel y as the correspondingpixel, and w_(n) represents the nth tap coefficient to be multipliedwith (the pixel value of) the nth low picture quality pixel. It is to benoted that, in the expression (1), it is assumed that the prediction tapis configured from N low picture quality pixels x₁, x₂, . . . and x_(N).

Here, the pixel value y of the high picture quality pixel can bedetermined not depending upon the linear primary expression indicated bythe expression (1) but by a high-order expression of the second- orhigher-order.

Here, if the true value of the pixel value of the high picture qualitypixel of a kth sample is represented by y_(k) and a prediction value ofthe true value y_(k) obtained by the expression (1) is represented byy_(k)′, then the prediction error e_(k) is represented by the followingexpression.[Math. 2]e _(k) =y _(k) −y _(k)′  (2)

Now, since the prediction value y_(k)′ of the expression (2) isdetermined in accordance with the expression (1), if y_(k)′ of theexpression (2) is rewritten in accordance with the expression (1), thenthe following expression is obtained.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 3} \right\rbrack & \; \\{e_{k} = {y_{k} - \left( {\sum\limits_{n = 1}^{N}{w_{n}x_{n,k}}} \right)}} & (3)\end{matrix}$

However, in the expression (3), x_(n,k) represents the nth low picturequality pixel configuring the prediction tap with respect to the highpicture quality pixel of the kth sample as the corresponding pixel.

Although the tap coefficient w_(n) with which the prediction error e_(k)of the expression (3) (or expression (2)) becomes 0 is optimum forprediction of the high picture quality pixel, it is generally difficultto determine such a tap coefficient w_(n) as just described in regard toall of the high picture quality pixels.

Therefore, if, for example, a least squares method is adopted as a normrepresenting that the tap coefficient w_(n) is optimum, then an optimumtap coefficient w_(n) can be determined by minimizing the sum total E(statistical errors) of square errors represented by the followingexpression.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 4} \right\rbrack & \; \\{E = {\sum\limits_{k = 1}^{K}e_{k}^{2}}} & (4)\end{matrix}$

It is to be noted that, in the expression (4), K represents a samplenumber (number of samples for learning) of a set of the high picturequality pixel y_(k) as the corresponding pixel and the low picturequality pixels x_(1,k), x_(2,k), . . . and x_(N,k) configuring theprediction tap with respect to the high picture quality pixel y_(k).

The lowest value (minimum value) of the sum total E of square errors ofthe expression (4) is given by w_(n) with which a value when the sumtotal E is partially differentiated with the tap coefficient w_(n) ismade 0 as given by the expression (5).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 5} \right\rbrack & \; \\{\frac{\partial E}{\partial w_{n}} = {{{e_{1}\frac{\partial e_{1}}{\partial w_{n}}} + {e_{2}\frac{\partial e_{2}}{\partial w_{n}}} + \ldots + {e_{k}\frac{\partial e_{k}}{\partial w_{n}}}} = {0\mspace{14mu}\left( {{n = 1},2,\ldots\mspace{14mu},N} \right)}}} & (5)\end{matrix}$

Therefore, if the expression (3) given above is partially differentiatedwith the tap coefficient W_(n), then the following expression isobtained.

$\begin{matrix}{\mspace{79mu}\left\lbrack {{Math}.\mspace{14mu} 6} \right\rbrack} & \; \\{{\frac{\partial e_{k}}{\partial w_{1}} = {- x_{1,k}}},{\frac{\partial e_{k}}{\partial w_{2}} = {- x_{2,k}}},\ldots\mspace{14mu},{\frac{\partial e_{k}}{\partial w_{N}} = {- x_{N,k}}},\left( {{k = 1},2,\ldots\mspace{14mu},K} \right)} & (6)\end{matrix}$

The following expression is obtained from the expressions (5) and (6).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 7} \right\rbrack & \; \\{{{\sum\limits_{k = 1}^{K}{e_{k}x_{1,k}}} = 0},{{\sum\limits_{k = 1}^{K}{e_{k}x_{2,k}}} = 0},{{\ldots\mspace{14mu}{\sum\limits_{k = 1}^{K}{e_{k}x_{N,k}}}} = 0}} & (7)\end{matrix}$

The expression (7) can be represented by a normal equation representedby the expression (8) by substituting the expression (3) into e_(k) ofthe expression (7).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 8} \right\rbrack & \; \\{{\begin{bmatrix}\left( {\sum\limits_{k = 1}^{K}{x_{1,k}x_{1,k}}} \right) & \left( {\sum\limits_{k = 1}^{K}{x_{1,k}x_{2,k}}} \right) & \ldots & \left( {\sum\limits_{k = 1}^{K}{x_{1,k}x_{N,k}}} \right) \\\left( {\sum\limits_{k = 1}^{K}{x_{2,k}x_{1,k}}} \right) & \left( {\sum\limits_{k = 1}^{K}{x_{2,k}x_{2,k}}} \right) & \ldots & \left( {\sum\limits_{k = 1}^{K}{x_{2,k}x_{N,k}}} \right) \\\vdots & \vdots & \ddots & \vdots \\\left( {\sum\limits_{k = 1}^{K}{x_{N,k}x_{1,k}}} \right) & \left( {\sum\limits_{k = 1}^{K}{x_{N,k}x_{2,k}}} \right) & \ldots & \left( {\sum\limits_{k = 1}^{K}{x_{N,k}x_{N,k}}} \right)\end{bmatrix}\begin{bmatrix}w_{1} \\w_{2} \\\vdots \\w_{N}\end{bmatrix}} = {\quad\begin{bmatrix}\left( {\sum\limits_{k = 1}^{K}{x_{1,k}y_{k}}} \right) \\\left( {\sum\limits_{k = 1}^{K}{x_{2,k}y_{k}}} \right) \\\vdots \\\left( {\sum\limits_{k = 1}^{K}{x_{N,k}y_{k}}} \right)\end{bmatrix}}} & (8)\end{matrix}$

The normal equation of the expression (8) can be solved for the tapcoefficient w_(n) by using, for example, a sweeping method (eliminationmethod of Gauss-Jordan) or the like.

By establishing and solving the normal equation of the expression (8)for each class, an optimum tap coefficient (here, tap coefficient thatminimizes the sum total E of square errors) w_(n) can be determined foreach class.

FIG. 3 depicts a configuration example of a learning apparatus thatperforms learning for establishing and solving a normal equation of theexpression (8) to determine the tap coefficient W_(n).

Referring to FIG. 3, the learning apparatus 30 includes a teacher datageneration section 31, a student data generation section 32 and alearning section 33.

A learning image to be used for learning of the tap coefficient w_(n) issupplied to the teacher data generation section 31 and the student datageneration section 32. As the learning image, for example, a highpicture quality image having a high resolution can be used.

The teacher data generation section 32 generates a teacher image to beused as a mapping destination of mapping as prediction arithmeticoperation in accordance with the expression (1) as teacher data to beused as a teacher (true value) of learning of the tap coefficient,namely, teacher data to be obtained by a classification adaptiveprocess, and supplies the generated teacher image to the learningsection 33. Here, the teacher data generation section 32 supplies, forexample, a high picture quality pixel as the learning image as it is asthe teacher image to the learning section 33.

The student data generation section 32 generates, from a learning image,a student image to be made a conversion target by mapping as predictionarithmetic operation in accordance with the expression (1) as studentdata to be used as a student of learning of a tap coefficient, namely,as student data to be made a target of prediction arithmetic operationwith a tap coefficient, and supplies the generated student image to thelearning section 33. Here, the student data generation section 32performs, for example, filtering of the high picture quality image asthe learning image with an LPF (Low Pass Filter) to decrease theresolution to generate a low picture quality image, and supplies the lowpicture quality image as the student image to the learning section 33.

The learning section 33 successively determines pixels configuring thestudent image as student data from the student data generation section32 as a noticed pixel, and selects, in regard to the noticed pixels,pixels having a tap structure same as that selected by the tap selectionsection 21 of FIG. 2 as a prediction tap from the student image.Further, the learning section 33 establishes and solves a normalequation of the expression (8) for each class using the correspondingpixel configuring the teacher image corresponding to the noticed pixeland the prediction tap of the noticed pixel to determine tapcoefficients for each class.

FIG. 4 is a block diagram depicting a configuration example of thelearning section 33 of FIG. 3.

Referring to FIG. 4, the learning section 33 includes tap selectionsections 41 and 42, a classification section 43, an addition section 44and a coefficient calculation section 45.

A student image is supplied to the tap selection sections 41 and 42while a teacher image is supplied to the addition section 44.

The tap selection section 41 successively selects pixels configuring thestudent image as a noticed pixel and supplies information representingthe noticed pixel to necessary blocks.

Further, the tap selection section 41 selects, in regard to the noticedpixel, pixels same as those selected by the tap selection section 21 ofFIG. 2 from among the pixels configuring the student image to aprediction tap, and, as a result, obtains a prediction tap having a tapstructure same as that obtained by the tap selection section 21. Then,tap selection section 41 supplies the obtained prediction tap to theaddition section 44.

The tap selection section 42 selects, in regard to the noticed pixel,pixels same as those selected by the tap selection section 22 of FIG. 2to a prediction tap from among the pixels configuring the student image,and, as a result, obtains a class tap having a tap structure same asthat obtained by the tap selection section 22. Then, the tap selectionsection 42 supplies the obtained class tap to the classification section43.

The classification section 43 performs classification same as that ofthe classification section 23 of FIG. 2 using the class tap from the tapselection section 42 and outputs a class code corresponding to a classof the noticed pixel obtained as a result of the classification to theaddition section 44.

The addition section 44 acquires (a pixel value of) the correspondingpixel corresponding to the noticed pixel from the pixels configuring theteacher image and performs addition whose target is the correspondingpixel and (the pixel values of) the pixels of the student imageconfiguring the prediction tap regarding the noticed pixel supplied fromthe tap selection section 41 for each class code supplied from theclassification section 43.

In particular, a corresponding pixel y_(k) of the teacher image asteacher data, a prediction tap x_(n), of the noticed pixel as studentdata and a class code representing the class of the noticed pixel aresupplied to the addition section 44.

The addition section 44 performs, for each class of the noticed pixel,multiplication (x_(n,k)x_(n′,k)) of the student data in the matrix onthe left side of the expression (8) and arithmetic operation equivalentto summation (E) using the prediction tap (student data) x_(n,k).

Further, the addition section 44 also performs, using the prediction tap(student data) x_(n,k) and the teacher data y_(k) for each class of thenoticed pixel, multiplication (x_(n,k)y_(k)) of the student data x_(n,k)and the teacher data y_(k) in the vector on the right side of theexpression (8) and arithmetic operation equivalent to summation (Σ).

In particular, the addition section 44 has stored in a built-in memory(not depicted) thereof the component (Σx_(n,k)x_(n′,k)) of the matrix onthe left side and the component (Σx_(n,k)y_(k)) of the vector on theright side of the expression (8) determined in regard to thecorresponding pixel corresponding to the noticed pixel as teacher datain the preceding operation cycle, and adds a corresponding componentx_(n,k+1)x_(n′,k+1) or x_(n,k+1)y_(k+1) calculated using the teacherdata y_(k+1) and the student data x_(n,k+1) in regard to the teacherdata that has newly become a corresponding pixel corresponding to thenew noticed pixel to the component (Σx_(n,k)x_(n′,k)) of the matrix orthe component (Σx_(n,k)y_(k)) of the vector (performs additionrepresented by summation of the expression (8)).

Then, the addition section 44 performs the addition described abovesetting, for example, all of the pixels of the student image as anoticed pixel to establish a normal equation represented by theexpression (8) in regard to each class and then supplies the normalequation to the coefficient calculation section 45.

The coefficient calculation section 45 solves the normal equationregarding each class supplied from the addition section 44 to determinean optimum tap coefficient w_(n) for each class and supplies thedetermined optimum tap coefficients w_(n).

The tap coefficients w_(n) for the individual classes determined in sucha manner as described above can be stored into the coefficientacquisition section 24 in the image conversion apparatus 20 of FIG. 2.

FIG. 5 is a block diagram depicting a second configuration example ofthe image conversion apparatus that performs a classification adaptiveprocess.

It is to be noted that, in FIG. 5, like elements to those in FIG. 2 aredenoted by like reference characters and description thereof is suitablyomitted in the following description.

Referring to FIG. 5, the image conversion apparatus 20 includes the tapselection sections 21 and 22, classification section 23, coefficientacquisition section 24 and prediction arithmetic operation section 25.

Accordingly, the image conversion apparatus 20 of FIG. 5 is configuredsimilarly to that of FIG. 2.

However, in FIG. 5, the coefficient acquisition section 24 stores a seedcoefficient hereinafter described. Further, in FIG. 5, a parameter z issupplied from the outside to the coefficient acquisition section 24.

The coefficient acquisition section 24 generates a tap coefficient foreach class corresponding to the parameter z from the seed coefficient,acquires a tap coefficient of the class from the classification section23 from the tap coefficients for the individual classes, and suppliesthe acquired tap coefficients to the prediction arithmetic operationsection 25.

Here, while the coefficient acquisition section 24 in FIG. 2 stores thetap coefficients as they are, the coefficient acquisition section 24 inFIG. 5 stores the seed coefficient. From the seed coefficient, tapcoefficients can be generated by applying (determining) the parameter z.From such a point of view, the seed coefficient can be regarded asinformation equivalent to the tap coefficients.

FIG. 6 is a block diagram depicting a configuration example of alearning apparatus that performs learning of a seed coefficient to bestored into the coefficient acquisition section 24.

Here, for example, similarly as in the case described with reference toFIG. 3, it is conceived that, determining that an image having highpicture quality (high picture quality image) is a second image andanother image having low picture quality (low picture quality image)obtained by decreasing the spatial resolution of the high picturequality image is a first image, a prediction tap is selected from withinthe low picture quality image and the pixel value of a high picturequality pixel that is a pixel of the high picture quality image isdetermined (predicted) using the prediction tap and the tap coefficient,for example, by the linear primary prediction arithmetic operation ofthe expression (1).

Here, it is assumed that the tap coefficient w is generated by thefollowing expression using the seed coefficient and the parameter z.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 9} \right\rbrack & \; \\{w_{n} = {\sum\limits_{m = 1}^{M}{\beta_{m,n}z^{m - 1}}}} & (9)\end{matrix}$

It is to be noted that, in the expression (9), β_(m,n) represents an mthseed coefficient used for determination of the nth tap coefficientw_(n). It is to be noted that, in the expression (9), the tapcoefficient w_(n) is determined using M seed coefficients β_(1,n),β_(2,n), . . . and β_(M,n).

Here, the expression for determining the tap coefficient w_(n) from theseed coefficient β_(m,n) and the parameter z is not limited to theexpression (9).

Now, a value z^(m-1) that depends upon the parameter z in the expression(9) is defined by the following expression introducing a new variablet_(m).[Math. 10]t _(m) =z ^(m-1) (m=1,2, . . . ,M)  (10)

The following expression is obtained by substituting the expression (10)into the expression (9).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 11} \right\rbrack & \; \\{w_{n} = {\sum\limits_{m = 1}^{M}{\beta_{m,n}t_{m}}}} & (11)\end{matrix}$

According to the expression (11), the tap coefficient w_(n) isdetermined by a linear primary expression of the seed coefficientβ_(m,n) and the variable t_(m).

Incidentally, if the true value of the pixel value of the high picturequality pixel of the kth sample is represented as y_(k) and theprediction value of the true value y_(k) obtained by the expression (1)is represented as y_(k)′, then the prediction error e_(k) is representedby the following expression.[Math. 12]e _(k) =y _(k) −y _(k′)  (12)

Now, since the prediction value y_(k)′ of the expression (12) isdetermined in accordance with the expression (1), if y_(k)′ of theexpression (12) is replaced in accordance with the expression (1), thenthe following expression is obtained.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 13} \right\rbrack & \; \\{e_{k} = {y_{k} - \left( {\sum\limits_{n = 1}^{N}{w_{n}x_{n,k}}} \right)}} & (13)\end{matrix}$

It is to be noted that, in the expression (13), x_(n,k) represents annth low picture quality pixel configuring the prediction tap in regardto the high picture quality pixel of the kth sample as the correspondingpixel.

By substituting the expression (11) into w_(n) of the expression (13),the following expression is obtained.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 14} \right\rbrack & \; \\{e_{k} = {y_{k} - \left( {\sum\limits_{n = 1}^{N}{\left( {\sum\limits_{m = 1}^{M}{\beta_{m,n}t_{m}}} \right)x_{n,k}}} \right)}} & (14)\end{matrix}$

While the seed coefficient β_(m,n) with which the prediction error e_(k)of the expression (14) is made 0 is optimum for prediction of the highpicture quality pixel, it is generally difficult to determine such aseed coefficient β_(m,n) as described above for all high picture qualitypixels.

Therefore, if, for example, a minimal square method is adopted as a normrepresenting that the seed coefficient β_(m,n) is optimum, then anoptimum seed coefficient β_(m,n) can be determined by minimizing the sumtotal E (total errors) of square errors represented by the followingexpression.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 15} \right\rbrack & \; \\{E = {\sum\limits_{k = 1}^{K}e_{k}^{2}}} & (15)\end{matrix}$

It is to be noted that, in the expression (15), K represents the samplenumber (number of samples for learning) of a set of the high picturequality pixel y_(k) as the corresponding pixel and the low picturequality pixels x_(1,k), x_(2,k), . . . and x_(N,k) configuring theprediction tap with respect to the high picture quality pixel y_(k).

A minimum value (lowest value) of the sum total E of square errors ofthe expression (15) is given by β_(m,n) with which a result obtained bypartial differentiation of the sum total E with the seed coefficient m,nis made 0 as indicated by the expression (16).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 16} \right\rbrack & \; \\{{\frac{\partial E}{\partial\beta_{m,n}} = {\sum\limits_{k = 1}^{K}2}},\frac{\partial e_{k}}{\partial\beta_{m,n}},{e_{k} = 0}} & (16)\end{matrix}$

By substituting the expression (13) into the expression (16), thefollowing expression is obtained.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 17} \right\rbrack\mspace{14mu}} & \; \\{{\sum\limits_{k = 1}^{K}{t_{m}x_{n,k}e_{k}}} = {\sum\limits_{k = 1}^{K}{t_{m}{x_{n,k}\left( {{y_{k} - \left( {\sum\limits_{n = 1}^{N}{\left( {\sum\limits_{m = 1}^{M}{\beta_{m,n}t_{m}}} \right)x_{n,k}}} \right)} = 0} \right.}}}} & (17)\end{matrix}$

Now, X_(i,p,j,q) and Y_(i,p) are defined as indicated by the followingexpressions (18) and (19), respectively.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 18} \right\rbrack\mspace{14mu}} & \; \\{X_{i,p,j,q} = {\sum\limits_{k = 1}^{K}{x_{i,k}t_{p}x_{j,k}{t_{q}\left( {{i = 1},2,\ldots\mspace{14mu},{{N\text{:}\mspace{14mu} j} = 1},2,\ldots\mspace{14mu},{{N\text{:}\mspace{14mu} p} = 1},2,\ldots\mspace{14mu},{{M\text{:}\mspace{14mu} q} = 1},2,\ldots\mspace{14mu},M} \right)}}}} & (18) \\{\left\lbrack {{Math}.\mspace{14mu} 19} \right\rbrack\mspace{14mu}} & \; \\{Y_{i,p} = {\sum\limits_{k = 1}^{K}{x_{i,k}t_{p}y_{k}}}} & (19)\end{matrix}$

In this case, the expression (17) can be represented by a normalequation indicated by the expression (20) using X_(i,p,j,q) and Y_(i,p).

$\left\lbrack {{Math}.\mspace{14mu} 20} \right\rbrack\mspace{11mu}\begin{matrix}{\begin{bmatrix}X_{1,1,1,1} & X_{1,1,1,2} & \ldots & X_{1,1,1,M} & X_{1,1,2,1} & \ldots & X_{1,1,N,M} \\X_{1,2,1,1} & X_{1,2,1,2} & \ldots & X_{1,2,1,M} & X_{1,2,2,1} & \ldots & X_{1,2,N,M} \\\vdots & \vdots & \ddots & \vdots & \vdots & \; & \vdots \\X_{1,M,1,1} & X_{1,M,1,2} & \ldots & X_{1,M,1,M} & X_{1,M,2,1} & \ldots & X_{1,M,N,M} \\X_{2,1,1,1} & X_{2,1,1,2} & \ldots & X_{2,M,1,M} & X_{2,M,2,1} & \ldots & X_{2,M,N,M} \\\vdots & \vdots & \; & \vdots & \vdots & \ddots & \vdots \\X_{N,M,1,1} & X_{N,M,1,2} & \ldots & X_{N,M,1,M} & X_{N,M,2,M} & \ldots & X_{N,M,N,M}\end{bmatrix}{\quad\mspace{461mu}{\left\lbrack \begin{matrix}\beta_{1,1} \\\beta_{2,1} \\\vdots \\\beta_{M,1} \\\beta_{1,2} \\\vdots \\\beta_{M,N}\end{matrix} \right\rbrack = \begin{bmatrix}Y_{1,1} \\Y_{1,2} \\\vdots \\Y_{1,M} \\Y_{2,1} \\\vdots \\Y_{N,M}\end{bmatrix}}}} & (20)\end{matrix}$

The normal equation of the expression (20) can be solved for the seedcoefficient β_(m,n) by using, for example, a sweeping method or thelike.

In the image conversion apparatus 20 of FIG. 5, the seed coefficientβ_(m,n) for each class determined by performing learning forestablishing and solving a normal equation of the expression (20) foreach class using a great number of high picture quality pixels y₁, y₂, .. . and y_(k) as the teacher data and the low picture quality pixelsx_(1,k), x_(2,k), . . . and x_(N,k) configuring the prediction tap withrespect to each high picture quality pixel y_(k) as the student data isstored into the coefficient acquisition section 24. Then, in thecoefficient acquisition section 24, the tap coefficient w_(n) for eachclass is generated in accordance with the expression (9) from the seedcoefficient β_(m,n) and the parameter z given from the outside, and, inthe prediction arithmetic operation section 25, calculation of theexpression (1) is performed using the tap coefficient w_(n) and the lowpicture quality pixels (pixels of the first image) x_(n) configuring theprediction taps regarding the noticed pixel to determine (predictionvalue close to) the pixel value of the high picture quality pixel(corresponding pixel of the second image).

FIG. 6 depicts a configuration example of a learning apparatus thatestablishes and solves a normal equation of the expression (20) toperform learning for determining the seed coefficient β_(m,n) for eachclass.

It is to be noted that, in FIG. 6, like elements to those in FIG. 3 aredenoted by like reference characters and description of them is suitablyomitted below.

Referring to FIG. 6, the learning apparatus 30 includes the teacher datageneration section 31, a parameter generation section 61, a student datageneration section 62 and a learning section 63.

Accordingly, the learning apparatus 30 of FIG. 6 is common to that ofFIG. 3 in that it includes the teacher data generation section 31.

However, the learning apparatus 30 of FIG. 6 is different from that ofFIG. 3 in that it newly includes the parameter generation section 61.Further, the learning apparatus 30 of FIG. 6 is different from that ofFIG. 3 in that it includes the student data generation section 62 andthe learning section 63 in place of the student data generation section32 and the learning section 33, respectively.

The parameter generation section 61 generates and supplies severalvalues within a range that can be taken by the parameter z to thestudent data generation section 62 and the learning section 63.

For example, if it is assumed that the value that can be taken by theparameter z is a real number of a range of 0 to z, then the parametergeneration section 61 generates, for example, parameters z having valuesof z=0, 1, 2, . . . and Z and supplies the parameters z to the studentdata generation section 62 and the learning section 63.

A learning image similar to that supplied to the teacher data generationsection 31 is supplied to the student data generation section 62.

The student data generation section 62 generates a student image fromthe learning image similarly to the student data generation section 32of FIG. 3 and supplies the student image as student data to the learningsection 63.

Here, several values within the range that can be taken by the parameterz are supplied from the parameter generation section 61 to the studentdata generation section 62.

The student data generation section 62 performs filtering of a highpicture quality image as a learning image, for example, by an LPF havinga cutoff frequency corresponding to the parameter z supplied thereto togenerate a low picture quality image as a student image with regard toeach of the several values of the parameter z.

In particular, in the student data generation section 62, Z+1 kinds oflow picture quality images as student images among which the spatialresolution is different are generated in regard to the high picturequality image as the learning image.

It is to be noted that it is assumed that the high picture quality imageis filtered to generate low picture quality images as student imagesusing, for example, an LPF whose cutoff frequency increases as the valueof the parameter z increases. In this case, the low picture qualityimage as a student image corresponding to the parameter z having a highvalue has a high spatial resolution.

Further, the student data generation section 62 can generate a lowpicture quality image as a student image in which the spatial resolutionin one of or both the horizontal direction and the vertical direction ofthe high picture quality image as a learning image is decreased inresponse to the parameter z can be generated.

Furthermore, in the case where a low picture quality image as a studentimage in which the spatial resolution in both the horizontal directionand the vertical direction of the high picture quality image as alearning image is to be generated, the spatial resolutions in thehorizontal direction and the vertical direction of the high picturequality image as a learning image can be decreased independently of eachother in response to parameters different from each other, namely, inresponse to two parameters z and z′.

In this case, in the coefficient acquisition section 24 of FIG. 5, twoparameters z and z′ are given from the outside and tap coefficients aregenerated using the two parameters z and z′ and the seed coefficient.

As described above, a seed coefficient from which tap coefficients canbe generated can be determined using, as the seed coefficient, the oneparameter z, the two parameters z and z′ or the three or moreparameters. However, in the present specification, for the convenienceof simplified description, description is given taking a seedcoefficient from which tap coefficients are generated using a singleparameter z as an example.

The learning section 63 determines a seed coefficient for each classusing a teacher image as teacher data from the teacher data generationsection 31, a parameter z from the parameter generation section 61 and astudent image as student data from the student data generation section62, and outputs the seed coefficients for the individual classes.

FIG. 7 is a block diagram depicting a configuration example of thelearning section 63 of FIG. 6.

It is to be noted that, in FIG. 7, like elements to those of thelearning section 33 in FIG. 3 are denoted by like reference charactersand description of them is suitably omitted below.

Referring to FIG. 7, the learning section 63 includes the tap selectionsections 41 and 42, the classification section 43, an addition section71 and a coefficient calculation section 72.

Accordingly, the learning section 63 of FIG. 7 is common to the learningsection 33 of FIG. 4 in that it includes the tap selection sections 41and 42 and the classification section 43.

However, the learning section 63 is different from the learning section33 in that it includes the addition section 71 and the coefficientcalculation section 72 in place of the addition section 44 and thecoefficient calculation section 45, respectively.

In FIG. 7, the tap selection sections 41 and 42 select a prediction tapand a class tap from a student image (here, a low picture quality imageas student data generated using an LPF of a cutoff frequencycorresponding to the parameter z) generated corresponding to theparameter z generated by the parameter generation section 61.

The addition section 71 acquires a corresponding pixel corresponding toa noticed pixel from a teacher image from the teacher data generationsection 31 of FIG. 6 and performs addition whose target is thecorresponding pixel, student data (pixels of the student image)configuring a prediction tap configured in regard to the noticed pixelsupplied from the tap selection section 41 and parameter z upongeneration of the student data for each class supplied from theclassification section 43.

In particular, to the addition section 71, the teacher data y_(k) as thecorresponding pixel corresponding to the noticed pixel, prediction tapx_(i,k) (x_(j,k)) relating to the noticed pixel outputted from the tapselection section 41 and class of the noticed pixel outputted from theclassification section 43 are supplied and the parameter z when thestudent data configuring the prediction tap regarding the noticed pixelis generated is supplied from the parameter generation section 61.

Then, the addition section 71 performs, using the prediction tap(student data) x_(i,k) (x_(j,k)) and the parameter z for each classsupplied from the classification section 43, multiplication(x_(i,k)t_(p)x_(j,k)t_(q)) of the student data and the parameter z fordetermining the component X_(i,p,j,q) defined by the expression (18) inthe matrix on the left side of the expression (20) and arithmeticoperation equivalent to the summation (Σ). It is to be noted that t_(p)of the expression (18) is calculated from the parameter z in accordancewith the expression (10). Also t_(q) of the expression (18) isdetermined similarly.

Further, the addition section 71 also performs, using the prediction tap(student data) x_(i,k), teacher data y_(k) and parameter z for eachclass supplied from the classification section 43, multiplication(x_(i,k)t_(p)y_(k)) of the student data x_(i,k), teacher data y_(k) andparameter z for determination of the component Y_(i,p) defined by theexpression (19) in the vector on the right side of the expression (20)and arithmetic operation corresponding to the summation (Σ). It is to benoted that t_(q) of the expression (19) is calculated from the parameterz in accordance with the expression (10).

In particular, the addition section 71 has stored in a built-in memorythereof (not depicted) the component X_(i,p,j,q) in the matrix on theleft side of the expression (20) determined, as the teacher data in thepreceding operation cycle, in regard to the corresponding pixelcorresponding to the noticed pixel and the component Y_(i,p) of thevector on the right side, and adds a corresponding componentx_(i,k)t_(p)x_(j,k)t_(q) or x_(i,k)t_(p)y_(k) calculated using theteacher data y_(k), student data x_(i,k) (x_(i,k)) and parameter z inregard to the teacher data that has become a corresponding pixelcorresponding to a new noticed pixel for the component X_(i,p,j,p) ofthe matrix or the component Y_(i,p) of the vector (performs additionrepresented by summation in the component X_(i,p,j,p) of the expression(18) or the component Y_(i,p) of the expression (19)).

Then, the addition section 71 establishes a normal equation indicated bythe expression (20) for each class by performing addition describedabove setting all pixels of the student image as the noticed pixel forall values of 0, 1, . . . and Z of the parameter z, and supplies thenormal equations to the coefficient calculation section 72.

The coefficient calculation section 72 solves the normal equation foreach class supplied from the addition section 71 to determine the seedcoefficient β_(m,n) for each class and outputs the seed coefficientsβ_(m,n).

Incidentally, while the learning apparatus 30 of FIG. 6 performslearning in which, determining a high picture quality image as alearning image as teacher data and setting a low picture quality imagethe spatial resolution of whose high picture quality image is degradedcorresponding to the parameter z as the student data, the seedcoefficient β_(m,n) that directly minimizes the sum total of squareerrors of the prediction value y of the teacher data predicted by alinear primary expression of the expression (1) from the tap coefficientw_(n) and the student data x_(n), as the learning of the seedcoefficient β_(m,n), learning for determining the seed coefficientβ_(m,n) that, as it were, indirectly minimize the sum total of squareerrors of the prediction value y of the teacher data can be performed.

In particular, using a high picture quality image as a learning image asteacher data and using, as student data, a low picture quality imagewhose horizontal resolution and vertical resolution are reduced byfiltering the high picture quality image by an LPF of a cutoff frequencycorresponding to the parameter z, at first a tap coefficient w_(n) thatminimizes the sum total of square errors of the prediction value y ofthe teacher data predicted by the linear primary prediction expressionof the expression (1) using the tap coefficient w_(n) and the studentdata x_(n) is determined for each value of the parameter z (here, z=0,1, . . . and Z). Then, using the tap coefficient w_(n) determined foreach value of the parameter z as teacher data and using the parameter zas student data, the seed coefficient β_(m,n) that minimizes the sumtotal of square errors of prediction values of the tap coefficient w_(n)as the teacher data predicted from the variable t_(m) corresponding tothe seed coefficient β_(m,n) and the parameter z that is the studentdata in accordance with the expression (11).

Here, the tap coefficient w_(n) that minimizes (makes minimum) the sumtotal E of square errors of the prediction value y of the teacher datapredicted by the linear primary prediction expression of the expression(1) can be determined for each value (z=0, 1, . . . and Z) of theparameter z for each class by establishing and solving a normal equationof the expression (8) similarly as in the case of the learning apparatus30 of FIG. 3.

Incidentally, the tap coefficient is determined from the seedcoefficient β_(m,n) and the variable t_(m) corresponding to theparameter z as indicated by the expression (11). Thus, if it is assumednow that the tap coefficient determined by the expression (11) isrepresented by w_(n)′, then the seed coefficient β_(m,n) with which theerror e_(n) between the optimum tap coefficient w_(n) and a tapcoefficient w_(n)′ determined by the expression (11), which isrepresented by the following expression (21), becomes 0 is a seedcoefficient optimum for determination of the optimum tap coefficientw_(n). However, it is generally difficult to determine such a seedcoefficient β_(m,n) as described above for all tap coefficients w_(n).[Math. 21]e _(n) =w _(n) −w _(n)′  (21)

It is to be noted that the expression (21) can be transformed into thefollowing expression with the expression (11).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 22} \right\rbrack & \; \\{e_{n} = {w_{n} - \left( {\sum\limits_{m = 1}^{M}{\beta_{m,n}t_{m}}} \right)}} & (22)\end{matrix}$

Thus, for example, if the minimum square method is adopted as a normrepresenting that the seed coefficient β_(m,n) is optimum, then theoptimum seed coefficient β_(m,n) can be determined by minimizing the sumtotal E (statistical errors) of square errors represented by thefollowing expression.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 23} \right\rbrack & \; \\{E = {\sum\limits_{n = 1}^{N}e_{n}^{2}}} & (23)\end{matrix}$

A minimum value (lowest value) of the sum total E of square errors ofthe expression (23) is given by β_(m,n) with which a result obtained bypartial differentiation of the sum total E with the seed coefficientβ_(m,n) is made 0 as indicated by the expression (24).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 24} \right\rbrack & \; \\{{\frac{\partial E}{\partial\beta_{m,n}} = {\sum\limits_{m = 1}^{M}{2\frac{\partial e_{n}}{\partial\beta_{m,n}}}}},{e_{n} = 0}} & (24)\end{matrix}$

The following expression is obtained by substituting the expression (22)by the expression (24).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 25} \right\rbrack & \; \\{{\sum\limits_{m = 1}^{M}{t_{m}\left( {w_{n} - \left( {\sum\limits_{m = 1}^{M}{\beta_{m,n}t_{m}}} \right)} \right)}} = 0} & (25)\end{matrix}$

Now, X_(i,j), and Y_(i) are defined as indicated by the expressions (26)and (27), respectively.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 26} \right\rbrack\mspace{14mu}} & \; \\{X_{i,j} = {\sum\limits_{z = 0}^{Z}{t_{i}{t_{j}\left( {{i = 1},2,\ldots\mspace{14mu},{{M\text{:}\mspace{14mu} j} = 1},2,\ldots\mspace{14mu},M} \right)}}}} & (26) \\{\left\lbrack {{Math}.\mspace{14mu} 27} \right\rbrack\mspace{14mu}} & \; \\{Y_{i} = {\sum\limits_{z = 0}^{Z}{t_{i}w_{n}}}} & (27)\end{matrix}$

In this case, the expression (25) can be represented by a normalequation indicated by the expression (28) using X_(i,j) and Y_(i).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 28} \right\rbrack & \; \\\begin{matrix}{{\begin{bmatrix}X_{1,1} & X_{1,2} & \ldots & X_{1,M} \\X_{2,1} & X_{2,1} & \ldots & X_{2,2} \\\vdots & \vdots & \ddots & \vdots \\X_{M,1} & X_{M,2} & \ldots & X_{M,M}\end{bmatrix}\left\lbrack \begin{matrix}\beta_{1,n} \\\beta_{2,n} \\\vdots \\\beta_{M,n}\end{matrix} \right\rbrack} = \begin{bmatrix}Y_{1} \\Y_{1} \\\vdots \\Y_{M}\end{bmatrix}} & \;\end{matrix} & (28)\end{matrix}$

Also the normal equation of the expression (28) can be solved for theseed coefficient β_(m,n), for example, by using a sweeping method or thelike.

FIG. 8 is a block diagram depicting a different configuration example ofthe learning section 63 of FIG. 6.

In particular, FIG. 8 depicts a configuration example of the learningsection 63 that establishes and solves a normal equation of theexpression (28) to determine a seed coefficient β_(m,n).

It is to be noted that, in FIG. 8, like elements to those in FIGS. 4 and7 are denoted by like reference characters and description of them isomitted suitably below.

The learning section 63 of FIG. 8 includes the tap selection sections 41and 42, the classification section 43, the coefficient calculationsection 45, addition sections 81 and 82 and a coefficient calculationsection 83.

Accordingly, the learning section 63 of FIG. 8 is common to the learningsection 33 of FIG. 4 in that it includes the tap selection sections 41and 42, classification section 43 and coefficient calculation section45.

However, the learning section 63 of FIG. 8 is different from thelearning section 33 of FIG. 4 in that it includes the addition section81 in place of the addition section 44 and newly includes the additionsection 82 and the coefficient calculation section 83.

A class of a noticed pixel outputted from the classification section 43and a parameter z outputted from the parameter generation section 61 aresupplied to the addition section 81. The addition section 81 performsaddition, whose target is teacher data as a corresponding pixelcorresponding to the noticed pixel from within a teacher image from theteacher data generation section 31 and student data configuring aprediction tap regarding the noticed pixel supplied from the tapselection section 41, for each class supplied from the classificationsection 43 and for each value of the parameter z outputted from theparameter generation section 61.

In particular, the teacher data y_(k), prediction tap x_(n,k), class ofthe noticed pixel and parameter z when a student image that configuresthe prediction tap x_(n,k) is generated are supplied to the additionsection 81.

The addition section 81 performs, using the prediction tap (studentdata) x_(n,k) for each class of the noticed pixel and for each value ofthe parameter z, multiplication (x_(n,k) x_(n′,k)) of the student datain the matrix on the left side of the expression (8) and arithmeticoperation equivalent to the summation (Σ).

Further, the addition section 81 performs, using the prediction tap(student data) x_(n,k) and the teacher data y_(k) for each class of thenoticed pixel and for each value of the parameter z, multiplication(x_(n,k) y_(k)) of the student data x_(n,k) and the teacher data y_(k)in the vector on the right side of the expression (8) and arithmeticoperation equivalent to the summation (Σ).

In particular, the addition section 81 stores the component(Σx_(n,k)x_(n′,k)) in the matrix of the left side and the component(Σx_(n,k)y_(k)) of the vector at the right side in the expression (8)determined relating to the corresponding pixel corresponding to thenoticed pixel at the last time as the teacher data in a built-in memory(not depicted), and adds a corresponding component x_(n,k+1)x_(n′,k+1)or x_(n,k+1)y_(k+1) determined using the teacher data y_(k)+1 and thestudent data x_(n,k+1) relating to the teacher data including thecorresponding pixel corresponding to a new noticed pixel to thecomponent (Σx_(n,k)x_(n′,k)) in the matrix or the component(Σx_(n,k)y_(k)) in the vector (performs addition represented by thesummation of the expression (8)).

Then, the addition section 81 performs addition described above usingall pixels of the student image as the noticed pixel to establish anormal equation indicated by the expression (8) for each value of theparameter z for each class and then supplies the normal equations to thecoefficient calculation section 45.

Accordingly, the addition section 81 establishes a normal equation ofthe expression (8) for each class similarly to the addition section 44of FIG. 4. However, the addition section 81 is different from theaddition section 44 in that it establishes a normal equation of theexpression (8) also for each value of the parameter z.

The coefficient calculation section 45 solves the normal equation foreach value of the parameter z for each class supplied from the additionsection 81 to determine a tap coefficient w_(n) optimum for each valueof the parameter z for each class and then supplies the tap coefficientsto the addition section 82.

The addition section 82 performs addition whose target is (the variablet_(m) corresponding to) the parameter z supplied from the parametergeneration section 61 (FIG. 6) and the optimum tap coefficient w_(n)supplied from the coefficient calculation section 45 for each class.

In particular, the addition section 82 performs, using the variablet_(i) (t_(j)) determined by the expression (10) from the parameter zsupplied from the parameter generation section 61, multiplication(t_(i)t_(j)) of the variables t_(i) (t_(j)) corresponding to theparameter z for determining the component X_(i,j) defined by theexpression (26) in the matrix on the left side of the expression (28)and arithmetic operation equivalent to the summation (Σ) for each class.

Here, since the component X_(i,j) depends only upon the parameter z andhas no relation to the class, actually calculation of the componentX_(i,j) need not be performed for each class but may be performed byonly once.

Further, the addition section 82 performs, using the variable t_(i)determined by the expression (10) from the parameter z supplied from theparameter generation section 61 and the optimum tap coefficient w_(n)supplied from the coefficient calculation section 45, multiplication(t_(i)w_(n)) of the variable t_(i) corresponding to the parameter z andthe optimum tap coefficient w_(n) for determining the component Y_(i)defined by the expression (27) in the vector on the right side of theexpression (28) and arithmetic operation equivalent to the summation (Σ)for each class.

The addition section 82 determines the component X_(i,j) represented bythe expression (26) and the component Y_(i) represented by theexpression (27) for each class to establish a normal equation of theexpression (28) for each class, and supplies the normal equations to thecoefficient calculation section 83.

The coefficient calculation section 83 solves the normal equation of theexpression (28) for each class supplied from the addition section 82 todetermine a seed coefficient β_(m,n) for each class and outputs the seedcoefficients β_(m,n).

It is possible for store the seed coefficients β_(m,n) for each classdetermined in such a manner as described above into the coefficientacquisition section 24 of FIG. 5.

It is to be noted that, also in the learning of a seed coefficient, itis possible to obtain seed coefficients for performing various imageconversion processes as the seed coefficient depending upon the mannerof selection of images to be made student data corresponding to a firstimage and teacher data corresponding to a second image similarly as inthe case of the learning of a tap coefficient.

In particular, in the case described above, learning of a seedcoefficient is performed using the learning image as it is as theteacher data corresponding to the second image and using the low picturequality image obtained by degrading the spatial resolution of thelearning image as the student data corresponding to the first image.Therefore, as the seed coefficient, a seed coefficient for performing aimage conversion process as the spatial resolution creation process forconverting the first image into the second image whose spatialresolution is improved can be obtained.

In this case, in the image conversion apparatus 20 of FIG. 5, thehorizontal resolution and the vertical resolution of an image can beimproved to the resolution corresponding to the parameter z.

Further, for example, by performing learning of a seed coefficientsetting a high picture quality image as teacher data and setting animage obtained by superimposing noise of a level corresponding to theparameter z with the high picture quality image as the teacher data asstudent data, as the seed coefficient, a seed coefficient for performingthe image conversion process as a noise removing process for convertingthe first image into the second image whose noise is removed (reduced)is obtained. In this case, in the image conversion apparatus 20 of FIG.5, an image having an S/N corresponding to the parameter z (image forwhich noise removal of a strength corresponding to the parameter z isperformed) can be obtained.

It is to be noted that, while, in the case described above, the tapcoefficient w_(n) is defined by β_(1,n)z⁰+β_(2,n)z¹+ . . .+β_(M,n)z^(M-1) as indicated by the expression (9) and the tapcoefficient w_(n) for improving both of the spatial resolutions in thehorizontal and vertical resolutions in accordance with the parameter zis determined by the expression (9), also it is possible to determine,as the tap coefficient w_(n), a tap coefficient that improves thehorizontal resolution and the vertical resolution independently of eachother in accordance with parameters z_(x) and z_(y) independent of eachother.

In particular, the tap coefficient w_(n) is defined, for example, by acubic polynomial β_(1,n)z_(x) ⁰z_(y) ⁰+β_(2,n)z_(x) ¹z_(y)⁰+β_(3,n)z_(x) ²z_(y) ⁰+β_(4,n)z_(x) ³z_(y) ⁰+β_(5,n)z_(x) ⁰z_(y)¹+β_(6,n)z_(x) ⁰z_(y) ²+β_(7,n)z_(x) ⁰z_(y) ³+β_(8,n)z_(x) ¹z_(y)¹+β_(9,n)z_(x) ²z_(y) ¹+β_(10,n)z_(x) ¹z_(y) ² in place of theexpression (9), and the variable t_(m) defined by the expression (10) isdefined, for example, as t₁=z_(x) ⁰z_(y) ⁰, t₂=z_(x) ¹z_(y) ⁰, t₃=z_(x)²z_(y) ⁰, t₄=z_(x) ³z_(y) ⁰, t₅=z_(x) ⁰z_(y) ¹, t₆=z_(x) ⁰z_(y) ²,t₁=z_(x) ⁰z_(y) ³, t₈=z_(x) ¹z_(y) ¹, t₉=z_(x) ²z_(y) ¹ and t₁₀=z_(x)¹z_(y) ² in place of the expression (10). Also in this case, the tapcoefficient w_(n) can be finally represented by the expression (11).Accordingly, in the learning apparatus 30 of FIG. 6, by performinglearning using, as student data, an image obtained by degrading thehorizontal resolution and the vertical resolution of teacher data inaccordance with the parameters z_(x) and z_(y) to determine the seedcoefficient β_(m,n), a seed coefficient β_(m,n) that improves thehorizontal resolution and the vertical resolution independently of eachother in accordance with the parameters z_(x) and z_(y) independent ofeach other can be determined.

Further, by further introducing the parameter z_(t) corresponding to aresolution in the time direction in addition to the parameters z_(x) andz_(y) corresponding to the horizontal resolution and the verticalresolution, respectively, tap coefficients w_(n) that improve thehorizontal resolution, vertical resolution and time resolutionindependently of each other in accordance with the parameters z_(x),z_(y) and z_(t) independent of each other can be determined.

Further, in the learning apparatus 30 of FIG. 6, by performing learningusing, as student data, an image obtained by degrading the horizontalresolution and the vertical resolution of teacher data in accordancewith the parameter z_(x) and adding noise to the teacher data inaccordance with the parameter z_(y) to determine a seed coefficientβ_(m,n), it is possible to determine the tap coefficient w_(n) thatimproves the horizontal resolution and the vertical resolution inaccordance with the parameter z_(x) and performs noise removal inaccordance with the parameter z_(y).

<First Configuration Example of Encoding Apparatus 11>

FIG. 9 is a block diagram depicting a first configuration example of theencoding apparatus 11 of FIG. 1.

Referring to FIG. 9, the encoding apparatus 11 includes an A/Dconversion section 101, a sorting buffer 102, an arithmetic operationsection 103, an orthogonal transform section 104, a quantization section105, a reversible encoding section 106 and an accumulation buffer 107.The encoding apparatus 11 further includes a dequantization section 108,an inverse orthogonal transform section 109, an arithmetic operationsection 110, a classification adaptive filter 111, a frame memory 112, aselection section 113, an intra-prediction section 114, a motionprediction compensation section 115, a prediction image selectionsection 116 and a rate controlling section 117.

The A/D conversion section 101 A/D converts an original image of ananalog signal into an original image of a digital signal and suppliesthe original image of a digital signal to the sorting buffer 102 so asto be stored into the sorting buffer 102.

The sorting buffer 102 sorts frames of the original image from adisplaying order to an encoding (decoding) order in response to the GOPs(Groups Of Pictures) and supplies the frames to the arithmetic operationsection 103, intra-prediction section 114, motion predictioncompensation section 115 and classification adaptive filter 111.

The arithmetic operation section 103 subtracts a prediction imagesupplied from the intra-prediction section 114 or the motion predictioncompensation section 115 through the prediction image selection section116 from the original image from the sorting buffer 102 and supplies aresidual (prediction residual) obtained by the subtraction to theorthogonal transform section 104.

For example, in the case of images for which inter encoding is to beperformed, the arithmetic operation section 103 subtracts a predictionimage supplied from the motion prediction compensation section 115 froman original image read out from the sorting buffer 102.

The orthogonal transform section 104 performs orthogonal transform suchas discrete cosine transform or Karhunen-Loeve transform for theresidual supplied from the arithmetic operation section 103. It is to benoted that the method for orthogonal transform is arbitrary. Theorthogonal transform section 104 supplies transform coefficientsobtained by the orthogonal transform to the quantization section 105.

The quantization section 105 quantizes the transform coefficientssupplied from the orthogonal transform section 104. The quantizationsection 105 sets a quantization parameter QP on the basis of a targetvalue for a code amount (code amount target value) supplied from therate controlling section 117 to perform quantization of the transformcoefficients. It is to be noted that the method for the quantization isarbitrary. The quantization section 105 supplies the quantized transformcoefficients to the reversible encoding section 106.

The reversible encoding section 106 encodes the transform coefficientsquantized by the quantization section 105 by a predetermined reversibleencoding method. Since the transform coefficients have been quantizedunder the control of the rate controlling section 117, the code amountof encoded data obtained by the reversible encoding of the reversibleencoding section 106 becomes equal to the code amount target value setby the rate controlling section 117 (or becomes close to the code amounttarget value).

Further, the reversible encoding section 106 acquires necessary encodinginformation from within encoding information relating to predictionencoding by the encoding apparatus 11 from the associated blocks.

Here, as the encoding information, for example, a prediction mode suchas intra prediction or inter prediction, motion information of a motionvector or the like, a code amount target value, a quantization parameterQP, a picture type (I, P, B), information of a CU (Coding Unit) or a CTU(Coding Tree Unit) and so forth are available.

For example, the prediction mode can be acquired from theintra-prediction section 114 or the motion prediction compensationsection 115. Further, for example, the motion information can beacquired from the motion prediction compensation section 115.

The reversible encoding section 106 acquires encoding information andacquires, from the classification adaptive filter 111, reductioninformation that reduces the tap number to be used for a classificationadaptive process by the classification adaptive filter 111.

The reversible encoding section 106 encodes (multiplexes) encodinginformation and reduction filter information by an arbitrary reversibleencoding method to form part of header information of encoded data.

The reversible encoding section 106 transmits encoded data through theaccumulation buffer 107. Accordingly, the reversible encoding section106 functions as a transmission section that transmits encoded data andeventually transmits encoding information and reduction filterinformation included in the encoded data.

As the reversible encoding method of the reversible encoding section106, for example, variable length coding, arithmetic coding or the likecan be adopted. As the variable length coding, for example, CAVLC(Context-Adaptive Variable Length Coding) prescribed in the H.264/AVCmethod and so forth are available. As the arithmetic coding, forexample, CABAC (Context-Adaptive Binary Arithmetic Coding) and so forthare available.

The accumulation buffer 107 temporarily accumulates encoded datasupplied from the reversible encoding section 106. The encoded dataaccumulated in the accumulation buffer 107 are read out and transmittedat a predetermined timing.

Transform coefficients quantized by the quantization section 105 aresupplied to the reversible encoding section 106 and supplied also to thedequantization section 108. The dequantization section 108 dequantizesthe quantized transform coefficients by a method corresponding toquantization by the quantization section 105. This dequantization methodmay be any method if it is a method corresponding to the quantizationprocess by the quantization section 105. The dequantization section 108supplies transform coefficients obtained by the dequantization to theinverse orthogonal transform section 109.

The inverse orthogonal transform section 109 inversely orthogonallytransforms the transform coefficients supplied from the dequantizationsection 108 by a method corresponding to the orthogonal transformprocess by the orthogonal transform section 104. This inverse orthogonaltransform method may be any method if it is a method corresponding tothe orthogonal transform process by the orthogonal transform section104. An inversely orthogonally transformed output (restored residual) issupplied to the arithmetic operation section 110.

The arithmetic operation section 110 adds a prediction image suppliedfrom the intra-prediction section 114 or the motion predictioncompensation section 115 through the prediction image selection section116 to the inverse orthogonal transform result supplied from the inverseorthogonal transform section 109, namely, to the restored residual, andoutputs a result of the addition as a decoding in-progress image that isin the process of decoding.

The decoding in-progress image outputted from the arithmetic operationsection 110 is supplied to the classification adaptive filter 111 or theframe memory 112.

The classification adaptive filter 111 performs a classificationadaptive process to perform an ILF process by a classification adaptiveprocess by a filter that functions as an ILF.

To the classification adaptive filter 111, not only a decodingin-progress image is supplied from the arithmetic operation section 110,but also an original image corresponding to the decoding in-progressimage is supplied from the sorting buffer 102 and necessary encodinginformation is supplied from the associated blocks of the encodingapparatus 11.

The classification adaptive filter 111 uses a student image equivalentto a decoding in-progress image from the arithmetic operation section110 and a teacher image equivalent to an original image from the sortingbuffer 102 and uses encoding information as occasion demands to performlearning for determining tap coefficients for each class.

In particular, the classification adaptive filter 111 uses, for example,a decoding in-progress image itself from the arithmetic operationsection 110 as a student image and uses an original image itself fromthe sorting buffer 102 as a teacher image to perform learning fordetermining tap coefficients for each class using encoding informationas occasion demands.

Further, the classification adaptive filter 111 performs a reductionprocess for reducing the tap coefficients for each class and generatesreduction filter information that reduces tap coefficients for eachclass by the reduction process. The reduction filter information issupplied from the classification adaptive filter 111 to the reversibleencoding section 106.

Further, the classification adaptive filter 111 uses tap coefficientsobtained using the reduction filter information to convert the decodingin-progress image from the arithmetic operation section 110 into anafter-filter image.

In particular, the classification adaptive filter 111 performs (imageconversion by) a classification adaptive process using the tapcoefficients for each class using the decoding in-progress image fromthe arithmetic operation section 110 as a first image and using encodinginformation as occasion demands to convert the decoding in-progressimage as the first image into an after-filter image as a second imageequivalent to the original image (generates an after-filter image) andoutputs the after-filter image.

The after-filter image outputted from the classification adaptive filter111 is supplied to the frame memory 112.

Here, in the classification adaptive filter 111, learning is performedusing a decoding in-progress image as a first image and using anoriginal image as a teacher image as described above, and tapcoefficients obtained by the leaning are used to perform aclassification adaptive process for converting the decoding in-progressimage into an after-filter image. Accordingly, the after-filter imageobtained by the classification adaptive filter 111 is an image veryclose to the original image.

The frame memory 112 temporarily stores a decoding in-progress imagesupplied from the arithmetic operation section 110 or an after-filterimage supplied from the classification adaptive filter 111 as a decodedimage decoded locally. The decoded image stored in the frame memory 112is supplied as a reference image to be used for generation of aprediction image to the selection section 113 at a necessary timing.

For example, from between a decoding in-progress image supplied from thearithmetic operation section 110 and an after-filter image supplied fromthe classification adaptive filter 111 as decoded images stored in theframe memory 112, the decoding in-progress image is used as a referenceimage for intra prediction. Meanwhile, the after-filter image is used asa reference image for inter prediction.

The selection section 113 selects a supply destination of a referenceimage supplied from the frame memory 112. For example, in the case whereintra prediction is to be performed by the intra-prediction section 114,the selection section 113 supplies the reference image supplied from theframe memory 112 to the intra-prediction section 114. On the other hand,for example, in the case where inter prediction is to be performed bythe motion prediction compensation section 115, the selection section113 supplies the reference image supplied from the frame memory 112 tothe motion prediction compensation section 115.

The intra-prediction section 114 uses an original image supplied fromthe sorting buffer 102 and a reference image supplied from the framememory 112 through the selection section 113 to perform intra prediction(in-screen prediction) basically using a PU (Prediction Unit) as a unitof processing. The intra-prediction section 114 selects an optimum intraprediction mode on the basis of a predetermined cost function andsupplies a prediction image generated by the optimum intra predictionmode to the prediction image selection section 116. Further, asdescribed hereinabove, the intra-prediction section 114 suitablysupplies a prediction mode indicative of the intra prediction modeselected on the basis of the cost function to the reversible encodingsection 106 and so forth.

The motion prediction compensation section 115 uses an original imagesupplied from the sorting buffer 102 and a reference image supplied fromthe frame memory 112 through the selection section 113 to perform motionprediction (inter prediction) basically using a PU as a unit ofprocessing. Further, the motion prediction compensation section 115performs motion compensation in response to a motion vector detected bythe motion prediction to generate a prediction image. The motionprediction compensation section 115 performs inter prediction in aplurality of inter prediction modes prepared in advance to generateprediction images.

The motion prediction compensation section 115 selects an optimum interprediction mode on the basis of a predetermined cost function for theprediction images obtained individually in regard to the interprediction modes. Further, the motion prediction compensation section115 supplies the prediction image generated by the optimum interprediction mode to the prediction image selection section 116.

Further, the motion prediction compensation section 115 supplies aprediction mode indicative of an inter prediction mode selected on thebasis of the cost function, motion information of a motion vector or thelike necessary upon decoding of encoded data encoded in the interprediction mode and so forth to the reversible encoding section 106.

The prediction image selection section 116 selects a supplying source ofthe prediction image (intra-prediction section 114 or motion predictioncompensation section 115) to be supplied to the arithmetic operationsections 103 and 110 and supplies a prediction image supplied from theselected supplying source to the arithmetic operation sections 103 and110.

The rate controlling section 117 controls the rate of quantizationoperation of the quantization section 105 on the basis of the codeamount of encoded data accumulated in the accumulation buffer 107 suchthat an overflow or an underflow does not occur. In particular, the ratecontrolling section 117 sets a target code amount for encoded data suchthat an overflow and an underflow of the accumulation buffer 107 do notoccur, and supplies the target code amount to the quantization section105.

<Configuration Example of Classification Adaptive Filter 111>

FIG. 10 is a block diagram depicting a configuration example of theclassification adaptive filter 111 depicted in FIG. 9.

Referring to FIG. 10, the classification adaptive filter 111 includes alearning apparatus 131, a reduction apparatus (reduction section) 132and an image conversion apparatus 133.

To the learning apparatus 131, an original image is supplied from thesorting buffer 102 (FIG. 9) and a decoding in-progress image is suppliedfrom the arithmetic operation section 110 (FIG. 9). Further, encodinginformation is applied to the learning apparatus 131.

The learning apparatus 131 uses the decoding in-progress image asstudent data and uses the original image as teacher data to performlearning for determining tap coefficients for each class (hereinafterreferred to as tap coefficient learning).

Further, the learning apparatus 131 supplies the tap coefficients foreach class obtained by the tap coefficient learning to the reductionapparatus 132.

It is to be noted that the learning apparatus 131 uses the encodinginformation as occasion demands to perform classification in the tapcoefficient learning.

The reduction apparatus 132 performs a reduction process for generatingreduction filter information that reduces the tap coefficients for eachclass from the learning apparatus 131 and supplies the reduction filterinformation obtained by the reduction process to the image conversionapparatus 133 and the reversible encoding section 106 (FIG. 9).

To the image conversion apparatus 133, a decoding in-progress image issupplied from the arithmetic operation section 110 (FIG. 9) andreduction filter information is supplied from the reduction apparatus132. Further, encoding information is supplied to the image conversionapparatus 133.

The image conversion apparatus 133 updates tap coefficients for eachclass using the reduction filter information of the reduction apparatus132. Further, the image conversion apparatus 133 uses the decodingin-progress process as a first image to perform image conversion by aclassification adaptive process using the tap coefficients for eachclass to convert the decoding in-progress image as the first image intoan after-filter image as a second image equivalent to an original image(generates an after-filter image), and supplies the after-filter imageto the frame memory 112 (FIG. 9).

It is to be noted that the image conversion apparatus 133 uses encodinginformation as occasion demands to perform classification in theclassification adaptive process.

<Example of Update Timing of Tap Coefficient of Image ConversionApparatus 133>

FIG. 11 is a view depicting an example of an update timing of tapcoefficients to be used for a classification adaptive process by theimage conversion apparatus 133.

The image conversion apparatus 133 updates tap coefficients, forexample, using a timing of after every F frames equal to or greater than1 frame as an update timing for updating tap coefficients to be used fora classification adaptive process as depicted in FIG. 11.

At the update timing, the learning apparatus 131 performs tapcoefficient learning to determine tap coefficients for each class. Here,a tap coefficient determined by the latest tap coefficient learning isreferred to as latest coefficient.

In the image conversion apparatus 133, at each update timing, tapcoefficients at present (hereinafter referred to as coefficients atpresent) used in the classification adaptive process are updated to thelatest coefficients.

In the encoding apparatus 11, in the case where the image conversionapparatus 133 updates the coefficients at present to the latestcoefficients, it is necessary also for the decoding apparatus 12(FIG. 1) to update coefficients at present to the latest coefficientssimilarly as in the encoding apparatus 11.

In order for the decoding apparatus 12 to update the coefficients atpresent to the latest coefficients, it is necessary, for example, totransmit the latest coefficients from the encoding apparatus 11 to thedecoding apparatus 12.

However, to transmit the latest coefficients of all classes from theencoding apparatus 11 to the decoding apparatus 12 at each update timingdegraded the compression efficiency.

Therefore, in the encoding apparatus 11, the reduction apparatus 132performs a reduction process for generating reduction filter informationthat reduces the latest coefficients for each class from the learningapparatus 131 and transmits the reduction filter information obtained bythe reduction process to the decoding apparatus 12 to improve thecompression efficiency.

<Example of Reduction Process of Reduction Apparatus 132>

FIG. 12 is a view illustrating an example of a reduction process of thereduction apparatus 132.

In FIG. 12, timings of frames after every F frames like the 0th frame,Fth frame, 2Fth frame, 3Fth frame, . . . are update timings.

At each update timing, the learning apparatus 131 performs tapcoefficient learning to determine the latest coefficients for eachclass.

The reduction apparatus 132 selects the latest coefficients of classesof 0 class or more from among the latest coefficients for each classobtained by tap coefficient learning in a reduction process and outputsthe selection coefficients as reduction filter information.

For example, the reduction apparatus 132 determines a merit decisionvalue representative of a degree of a merit in the case where the latestcoefficients are used for (prediction arithmetic operation of theexpression (1) of) a classification adaptive process in place of thecoefficients at present for each class. Then, the reduction apparatus132 selects the latest coefficients of classes in regard to which themerit decision value is equal to or higher than a threshold value asselection coefficients.

In FIG. 12, the latest coefficients of C classes 0, 1, . . . , C−1 aredetermined by tap coefficient learning at update timings.

At the update timing for the 0th frame, namely, for example, at a timingat which encoding is started, the latest coefficients of all of the Cclasses 0 to C−1 are selected as selection coefficients.

Then, at the update timing for the Fth frame, from among the latestcoefficients of the classes 0 to C−1, the latest coefficients of theclass 3 are the latest coefficients of a class whose merit decisionvalue is equal to or higher than the threshold value and is selected asa selection coefficient. On the other hand, at the update timing for the2Fth frame, from among the latest coefficients for the classes 0 to C−1,the latest coefficients for the class 2 are the latest coefficients of aclass whose merit decision value is equal to or higher than thethreshold value and the selection coefficients are selected.

As described above, the reduction apparatus 132 selects, from among thelatest coefficients for the C classes 0 to C−1, the latest coefficientsof a class or classes whose merit decision value is equal to or higherthan the threshold value as selection coefficients and outputs theselection coefficients as reduction filter information.

Accordingly, since, at each update timing, from the encoding apparatus11 to the decoding apparatus 12, not the latest coefficients of all Cclasses 0 to C−1 but the latest coefficients for the selected classes asselection coefficients are transmitted as reduction filter information,the compression efficiency can be improved in comparison with that in analternative case in which the latest coefficients of all of the Cclasses 0 to C−1 are normally transmitted at an update timing.

It is to be noted that, in the case where the merit decision value ofthe latest coefficients of most classes from among the latestcoefficients of the C classes 0 to C−1 is equal to or higher than thethreshold value, namely, in the case where the number of classes of thelatest coefficients with regard to which the merit decision value isequal to or higher than the threshold value is equal to or greater thana predetermined number close to the total number C of classes, thereduction apparatus 132 does not select the latest coefficients ofclasses whose merit decision value is equal to or higher than thethreshold value as selection coefficients but can select the latestcoefficients of all of the C classes 0 to C−1 as selection coefficientsand output the selected selection coefficients as reduction filterinformation.

Further, as the merit decision value, for example, a value correspondingto the RD (Rate-Distortion) cost can be adopted. In particular, if aclass of tap coefficients whose merit decision value is to be determinedis set as noticed class to be noticed, then as the merit decision value(of the latest coefficients) of the noticed class, then the RD costitself in the case where the latest coefficients are used in regard tothe noticed class or a value representative of a degree by which the RDcost in the case where the latest coefficients are used is superior(difference between RD cost values or the like) in regard to the noticedclass can be adopted.

Further, as the merit decision value, for example, a value correspondingto an inter-coefficient distance between the latest coefficient of thenoticed class and the current coefficient in the tap coefficient spacehaving coefficient axes of N tap coefficients w₁, w₂, . . . , w_(N) ofone class, namely, for example, an inter-coefficient distance betweenthe latest coefficient and the current coefficient, can be adopted.

Further, as the merit decision value, for example, for the noticedclass, a value corresponding to the S/N of the after-filter imagedetermined using the latest coefficients can be adopted. In particular,as the merit decision value, for example, the S/N itself of theafter-filtered image determined using the latest coefficients in regardto the noticed class, or, a value representative of the degree by whichthe S/N of the after-filter image determined using the latestcoefficients in comparison with the S/N of the after-filter imagedetermined using the coefficients at present in regard to the noticedclass (difference between the S/N values) can be adopted.

Furthermore, as the merit decision value, for example, a valuecorresponding to the use frequency by which the tap coefficient (currentcoefficient) of the noticed class is used (for the prediction arithmeticoperation of the expression (1)) in the classification adaptive processcan be adopted. In particular, for example, the number of times by whichthe tap coefficients of the noticed block are used in the classificationadaptive process within the latest fixed period such as a period fromthe update timing in the preceding operation cycle to the update timingin the current operation cycle can be adopted.

From the point of view of improvement of the compression efficiency andimprovement of the S/N of the decoded image, it is desirable to adopt avalue corresponding to the RD cost as the merit decision value. However,in the case where the RD cost is adopted as the merit decision value, ahigh calculation cost is required for calculation of the merit decisionvalue. By adopting, as the merit decision value, a value corresponding,for example, to the inter-coefficient distance, S/N of the after-filterimage or use frequency described above, the calculation cost requiredfor calculation of the merit decision value can be reduced.

<Configuration Example of Learning Apparatus 131>

FIG. 13 is a block diagram depicting a configuration example of thelearning apparatus 131 of FIG. 10.

It is to be noted that, in FIG. 13, elements corresponding to those ofFIGS. 3 and 4 are denoted by like reference numerals and description ofthem is hereinafter omitted suitably.

Referring to FIG. 13, the learning apparatus 131 includes a learningsection 33. The learning section 33 includes tap selection sections 41and 42, a classification section 43, an addition section 44 and acoefficient calculation section 45.

Accordingly, the learning apparatus 131 of FIG. 13 is common to that ofFIG. 3 in that it includes the learning section 33. Further, thelearning apparatus 131 is common to that of FIG. 4 in that the learningsection 33 includes the components from the tap selection section 41 tothe coefficient calculation section 45.

However, the learning apparatus 131 of FIG. 13 is different from that ofFIG. 3 in that it includes neither of the teacher data generationsection 31 and the student data generation section 32.

Further, the learning apparatus 131 of FIG. 13 is different from that ofFIG. 4 in that, in the learning section 33, encoding information issupplied to the classification section 43.

In the learning apparatus 131 of FIG. 13, the classification section 23performs classification using (image characteristic amounts obtainedfrom) class taps or encoding information as occasion demands.

Further, in the learning apparatus 131 of FIG. 13, tap coefficientlearning is performed using a decoding in-progress image as student dataand using an original image corresponding to the decoding in-progressimage as teacher data, and tap coefficients for each class obtained bythe tap coefficient learning are supplied as the latest coefficientsfrom the coefficient calculation section 45 to the reduction apparatus132.

The tap coefficient learning by the learning apparatus 131 not only canbe performed in parallel to encoding of an original image but also canbe performed in advance irrespective of encoding of an original image.

In the case where the tap coefficient learning is performed in advanceirrespective of encoding of the original image, the tap coefficientlearning can be performed for each category using original images of aplurality of categories as teacher data. Then, at an update timing, forexample, the tap coefficients of a category that minimizes apredetermined cost function from among the tap coefficients of theplurality of categories can be outputted as the latest coefficients.

It is to be noted that the classification section 43 can performclassification using one or both of an image characteristic amount of anoticed pixel obtained from pixel values of the pixels in the decodingin-progress image as a class tap and encoding information of the noticedpixel.

As the class tap, for example, nine pixels of a cross shape centered atthe noticed pixel, nine pixels in a square of 3×3 pixels centered at thenoticed pixel, 13 pixels in a diamond shape centered at the noticedpixel and having diagonals in a horizontal direction and a verticaldirection or the like can be adopted.

Further, the class tap can be configured, for example, from pixels of anoticed frame that is a frame (picture) of the noticed pixel and pixelsof a frame other than the noticed frame such as a frame adjacent thenoticed pixel.

Further, as the image characteristic amount to be used forclassification, for example, the ADRC code (ADRC value) obtained by a1-bit ADRC process for the class tap can be adopted.

Now, if it is assumed that, for simplified description, theclassification section 43 performs classification of a noticed pixelusing only an ADRC code as an image characteristic amount, then theclassification section 43 classifies the noticed pixel, for example,into a class of a class code corresponding to an ADRC code.

For example, in the case where the class code is represented by adecimal number, the class code corresponding to the ADRC code signifiesa value obtained by converting, regarding the ADRC code as a binarynumber, the ADRC code of the binary number into a decimal number.

For example, if the ADRC code obtained by a 1-bit ADRC process for aclass tap configured from nine pixels is 000011100, then the class codecorresponding to the ADRC code 000011100 is 28.

Here, as the image feather value to be used for classification, inaddition to the ADRC code obtained from the class tap, an imagecharacteristic amount other than the ADRC code such as a dynamic range,difference absolute value sum or the like of pixel values of pixelsconfiguring the class tap can be adopted.

In the case where, for example, the dynamic range of pixel values ofpixels configuring the class tap is adopted as the image characteristicamount to be used for classification, for example, by thresholdprocessing the dynamic range with one or more threshold values, thenoticed pixel can be classified into one of two or more classes.

Furthermore, the classification can be performed not using one imagecharacteristic amount such as the ADRC mode but using a plurality ofimage characteristic amounts such as, for example, the ADRC code and adynamic range or the like.

Further, the classification can be performed using encoding informationof the noticed pixel in addition to an image characteristic amount ofthe noticed pixel.

As the encoding information of the noticed pixel to be used forclassification, for example, a block phase representative of theposition of the noticed pixel in a block such as a CU or a PU includingthe noticed pixel, a picture type of a picture including the noticepixel, a quantization parameter QP of a PU including the noticed pixelor the like can be adopted.

In the case where the block phase is adopted as the encoding informationof the noticed pixel to be used for classification, the noticed pixelcan be classified, for example, depending upon whether or not thenoticed pixel is a pixel on the boundary of a block.

On the other hand, in the case where the picture type is adopted as theencoding information of the noticed pixel to be used for classification,the noticed pixel can be classified, for example, depending upon whichone of an I picture, a P picture and a B picture the picture includingthe noticed pixel is.

Further, in the case where the quantization parameter QP is adopted asthe encoding information of the noticed pixel to be used forclassification, the noticed pixel can be classified, for example,depending upon the roughness (fineness) of quantization.

Further, classification can be performed not only using an imagecharacteristic amount or encoding information but also using both animage characteristic amount and encoding information.

<Configuration Example of Reduction Apparatus 132>

FIG. 14 is a block diagram depicting an example of a configurationexample of the reduction apparatus 132 of FIG. 10.

Referring to FIG. 14, the reduction apparatus 132 includes a selectionsection 141.

To the selection section 141, tap coefficients for each class as thelatest coefficients are supplied from the learning apparatus 131.

The selection section 141 determines a merit decision valuerepresentative of a degree of a merit in the case where the latestcoefficients are used for (prediction arithmetic operation of theexpression (1) of) a classification adaptive process in place of thecoefficients at present for each class of the latest coefficients fromthe learning apparatus 131.

Then, the selection section 141 selects the latest coefficients ofclasses in regard to which the merit decision value is equal to orhigher than a threshold value as selection coefficients and supplies theselection coefficients as reduction filter information to the imageconversion apparatus 133 (FIG. 10) and the reversible encoding section106 (FIG. 9).

<Configuration Example of Image Conversion Apparatus 133>

FIG. 15 is a block diagram depicting a configuration example of theimage conversion apparatus 133 of FIG. 10.

It is to be noted that, in FIG. 15, elements corresponding to those ofthe image conversion apparatus 20 of FIG. 2 are denoted by likereference numerals and description of them is hereinafter omittedsuitably.

Referring to FIG. 15, the image conversion apparatus 133 includes thecomponents from the tap selection section 21 to the classificationsection 23, the prediction arithmetic operation section 25 and acoefficient acquisition section 151.

Accordingly, the image conversion apparatus 133 is configured similarlyto the image conversion apparatus 20 of FIG. 2 in that it includes thecomponents from the tap selection section 21 to the classificationsection 23, and the prediction arithmetic operation section 25.

However, the image conversion apparatus 133 is different from the imageconversion apparatus 20 in that it includes the coefficient acquisitionsection 151 in place of the coefficient acquisition section 24.

In the image conversion apparatus 133, a decoding in-progress image issupplied as a first image to the tap selection sections 21 and 22, andthe prediction arithmetic operation section 25 determines anafter-filter image as a second image.

Further, in the image conversion apparatus 133, encoding information issupplied to the classification section 23, and the classificationsection 23 performs classification similar to that by the classificationsection 43 of the learning apparatus 131 (FIG. 13) using a class tap andthe encoding information as occasion demands.

To the coefficient acquisition section 151, selection coefficients asreduction filter information are supplied from the reduction apparatus132.

The coefficient acquisition section 151 uses the selection coefficientsas the reduction filter information from the reduction apparatus 132 toobtain tap coefficients for each class to be used for a classificationadaptive process. Then, the coefficient acquisition section 151 acquirestap coefficients of the class of the noticed pixel from the tapcoefficients for the individual classes obtained using the electioncoefficients as the reduction filter process and supplies the tapcoefficients to the prediction arithmetic operation section 25.

FIG. 16 is a block diagram depicting a configuration example of thecoefficient acquisition section 151 of FIG. 15.

Referring to FIG. 16, the coefficient acquisition section 151 includesan updating section 161, a storage section 162 and an acquisitionsection 163.

To the updating section 161, selection coefficients as reduction filterinformation are supplied from the reduction apparatus 132.

The updating section 161 updates tap coefficients for individual classesas coefficients at present stored in the storage section 162 with theselection coefficients as the reduction filter information from thereduction apparatus 132.

The storage section 162 stores the tap coefficients for the individualclasses.

Here, the storage section 162 is reset at a predetermined timing suchas, for example, a timing at which the power supply to the encodingapparatus 11 is turned on, a timing at which encoding of a sequence ofan original image is started in the encoding apparatus 11 or the like(the storage substance of the storage section 162 is initialized).

The timing for resetting of the storage section 162 (hereinafterreferred to as initialization timing) is an update timing of the tapcoefficients, and the learning apparatus 131 performs tap coefficientlearning and tap coefficients for each class as the latest coefficientsobtained by the tap coefficient learning are supplied to the reductionapparatus 132.

At the initialization timing, the reduction apparatus 132 selects tapcoefficients of all classes as the latest coefficients from the learningapparatus 131 as selection coefficients and outputs the selectioncoefficients as reduction filter information to the reversible encodingsection 106 (FIG. 9) and the updating section 161.

In this case, the updating section 161 stores the tap coefficients ofall classes as the reduction filter information from the reductionapparatus 132 into the storage section 162.

The image conversion apparatus 133 performs a classification adaptiveprocess using the tap coefficients for the individual classes stored inthe storage section 162 in such a manner as described above as thecoefficients at present.

Then, when an update timing comes thereafter and reduction filterinformation is supplied from the reduction apparatus 132 to the updatingsection 161, the updating section 161 updates the coefficients atpresent of the classes of the selection coefficients from among thecoefficients at present stored in the storage section 162 into theselection coefficients as the reduction filter information from thereduction apparatus 132 with the selection coefficients.

To the acquisition section 163, (a class code of) a class of the noticedpixel is supplied from the classification section 23. The acquisitionsection 163 acquires tap coefficients as the coefficients at present ofthe class of the noticed pixel from the coefficients at present storedin the storage section 162 and supplies the tap coefficients to theprediction arithmetic operation section 25.

<Encoding Process>

FIG. 17 is a flow chart illustrating an example of the encoding processof the encoding apparatus 11 of FIG. 9.

It is to be noted that the order of steps of the encoding processdepicted in FIG. 17 and so forth is an order for the convenience ofdescription, and steps of an actual encoding process are performedsuitably in parallel in a necessary order. This similarly applies alsoto the encoding process hereinafter described.

In the encoding apparatus 11, the learning apparatus 131 (FIG. 10) ofthe classification adaptive filter 111 temporarily stores a decodingin-progress image supplied thereto as student data and temporarilystores an original image corresponding to the decoding in-progress imageas teacher data.

Then at step S11, the learning apparatus 131 decides whether the timingat present is an update timing for tap coefficients.

Here, the update timing for tap coefficients can be determined inadvance such as a timing, for example, after every one or more frames(pictures), after every one or more sequences, after every one or moreslices, after every one or more line of a predetermined block such as aCTU or the like.

Further, as the update timing for tap coefficients, not only a periodic(fixed) timing such as a timing after one or more frames (pictures) butalso a so-called dynamic timing such as a timing at which the S/N of anafter-filter image becomes equal to or lower than a threshold value(timing at which the error of an after-filter image from an originalimage becomes equal to or greater than a threshold value), a timing atwhich the (absolute value sum or the like of) the residual becomes equalto or greater than a threshold value can be adopted.

In the case where it is decided at step S11 that the timing at presentis not an update timing for tap coefficients, the processing advances tostep S21 skipping steps S12 to S20.

On the other hand, in the case where it is decided at step S11 that thetiming at present is an update timing for tap coefficients, theprocessing advances to step S12, at which the learning apparatus 131performs tap coefficient learning.

In particular, the learning apparatus 131 performs tap coefficientlearning using an after-filter image and an original image that havebeen stored after an update timing in the preceding operation cycle tillan update timing in the current operation cycle to determine tapcoefficients as the latest coefficients for each class.

Then, the learning apparatus 131 supplies the latest coefficients foreach class obtained by the tap coefficient learning to the reductionapparatus 132, and the processing advances from step S12 to step S13.

At step S13, the selection section 141 of the reduction apparatus 132(FIG. 14) selects one class that has not been selected as the noticedclass from among all classes in regard to which tap coefficients are tobe determined by tap coefficient learning as a noticed class, and theprocessing advances to step S14.

At step S14, the selection section 141 calculates a merit decision valuesuch as, for example, an RD cost or the like in regard to the latestcoefficients of the noticed class, and the processing advances to stepS15.

At step S15, the selection section 141 decides whether the meritdecision value regarding the latest coefficients of the noticed class isequal to or higher than a threshold value determined in advance.

In the case where it is decided at step S15 that the merit decisionvalue regarding the latest coefficients of the noticed class is equal toor higher than the threshold value, the processing advances to step S16,at which the selection section 141 selects the latest coefficients ofthe noticed class as selection coefficients.

Then, the processing advances from step S16 to step S17, at which theselection section 141 outputs the selection coefficients as reductionfilter information to the reversible encoding section 106 (FIG. 9) andthe image conversion apparatus 133 (FIG. 10). Thereafter, the processingadvances to step S18.

On the other hand, in the case where it is decided at step S15 that themerit decision value regarding the latest coefficients of the noticedclass is not equal to or higher than the threshold value, the processingadvances to step S18 skipping steps S16 and S17.

Accordingly, the latest coefficients of the noticed class are supplied,only in the case where the merit decision value thereof is equal to orhigher than the threshold value, as reduction filter information to thereversible encoding section 106 (FIG. 9) and the image conversionapparatus 133 (FIG. 10).

At step S18, the selection section 141 of the reduction apparatus 132(FIG. 14) decides whether all classes whose tap coefficient is to bedetermined by tap coefficient learning have been determined as a noticedclass.

In the case where it is decided at step S18 that all classes have notbeen determined as a noticed class as yet, the processing returns tostep S13, whereafter similar processes are repeated.

On the other hand, at step S18, in the case where it is decided that allclasses have been determined as a noticed class, the processing advancesto step S19, at which the reversible encoding section 106 (FIG. 9) setsthe reduction filter information from the selection section 141 of thereduction apparatus 132 as a transmission target. Thereafter, theprocessing advances to step S20. The reduction filter process set as thetransmission target is included into and transmitted together withencoding data in a prediction encoding process that is performed at stepS21 hereinafter described.

At step S20, in the image conversion apparatus 133, the updating section161 of the coefficient acquisition section 151 (FIG. 16) uses theselection coefficients as the reduction filter information from thereduction apparatus 132 to update the tap coefficients of the class ofthe selection coefficients from among the coefficients at present storedin the storage section 162 from the coefficients at present to theselection coefficients, and the processing advances to step S21.

At step S21, a prediction encoding process for the original image isperformed, and the encoding process ends therewith.

FIG. 18 is a flow chart illustrating an example of the predictionencoding process at step S21 of FIG. 17.

In the prediction encoding process, at step S31, the A/D conversionsection 101 (FIG. 9) A/D converts the original image and supplies theoriginal image after the A/D conversion to the sorting buffer 102,whereafter the processing advances to step S32.

At step S32, the sorting buffer 102 stores the original image from theA/D conversion section 101, sorts the original image in an encodingorder and outputs the original image after the sorting, whereafter theprocessing advances to step S33.

At step S33, the intra-prediction section 114 performs anintra-prediction process of an intra-prediction mode, and the processingadvances to step S34. At step S34, the motion prediction compensationsection 115 performs an inter motion prediction process in which motionprediction and motion compensation in the inter-prediction mode areperformed, and the processing advances to step S35.

In the intra-prediction process of the intra-prediction section 114 andthe inter motion prediction process of the motion predictioncompensation section 115, a cost function for various prediction modesis arithmetically operated and a prediction image is generated.

At step S35, the prediction image selection section 116 determines anoptimum prediction mode on the basis of cost functions obtained by theintra-prediction section 114 and the motion prediction compensationsection 115. Then, the prediction image selection section 116 selectsand outputs a prediction image of an optimum prediction mode frombetween the prediction image generated by the intra-prediction section114 and the prediction image generated by the motion predictioncompensation section 115, and the processing advances from step S35 tostep S36.

At step S36, the arithmetic operation section 103 arithmeticallyoperates the residual between the target image of the encoding targetthat is the original image outputted from the sorting buffer 102 and theprediction image outputted from the prediction image selection section116 and outputs the residual to the orthogonal transform section 104,whereafter the processing advances to step S37.

At step S37, the orthogonal transform section 104 orthogonallytransforms the residual from the arithmetic operation section 103 andsupplies the transform coefficients obtained as a result of theorthogonal transform to the quantization section 105. Thereafter, theprocessing advances to step S38.

At step S38, the quantization section 105 quantizes the transformcoefficients from the orthogonal transform section 104 and suppliesquantization coefficients obtained by the quantization to the reversibleencoding section 106 and the dequantization section 108. Thereafter, theprocessing advances to step S39.

At step S39, the dequantization section 108 dequantizes the quantizationcoefficients from the quantization section 105 and supplies transformcoefficients obtained as a result of the dequantization to the inverseorthogonal transform section 109. Then, the processing advances to stepS40. At step S40, the inverse orthogonal transform section 109 inverselyorthogonally transforms the transform coefficients from thedequantization section 108 and supplies a residual obtained as a resultof the inverse orthogonal transform to the arithmetic operation section110. Thereafter, the processing advances to step S41.

At step S41, the arithmetic operation section 110 adds the residual fromthe inverse orthogonal transform section 109 and the prediction imageoutputted from the prediction image selection section 116 to generate adecoding in-progress image corresponding to the original image that hasbecome the target of the arithmetic operation of the residual by thearithmetic operation section 103. The arithmetic operation section 110supplies the decoding in-progress image to the classification adaptivefilter 111 or the frame memory 112, and the processing advances fromstep S41 to step S42.

In the case where the decoding in-progress image is supplied from thearithmetic operation section 110 to the classification adaptive filter111, at step S42, the classification adaptive filter 111 performs aclassification adaptive process (classification adaptive filter process)as an ILF process for the decoding in-progress image from the arithmeticoperation section 110. Since the classification adaptive process isperformed for the decoding in-progress image, an after-filter imagecloser to the original image than that in the case where the decodingin-progress image is filtered by an ILF is determined (generated).

The classification adaptive filter 111 supplies the after-filter imageobtained by the classification adaptive process to the frame memory 112,and the processing advances from step S42 to step S43.

At step S43, the frame memory 112 stores the after-filter image suppliedfrom the arithmetic operation section 110 or the after-filter imagesupplied from the classification adaptive filter 111 as a decoded image,and the processing advances to step S44. The decoded image stored in theframe memory 112 is used as a reference image on the basis of which aprediction image is to be generated as step S34 or S35.

At step S44, the reversible encoding section 106 encodes thequantization coefficients from the quantization section 105. Further,the reversible encoding section 106 encodes encoding information such asthe quantization parameter QP used in the quantization by thequantization section 105, the prediction mode obtained by theintra-prediction process by the intra-prediction section 114, theprediction mode or the motion information obtained by theintra-prediction process by the motion prediction compensation section115 and so forth as occasion demands, and places the encoded encodinginformation into encoded data.

Further, the reversible encoding section 106 encodes reduction filterinformation set as the transmission target at step S19 of FIG. 17 andplaces the encoded reduction filter information into the encoded data.Then, the reversible encoding section 106 supplies the encoded data tothe accumulation buffer 107, and the processing advances from step S44to step S45.

At step S45, the accumulation buffer 107 accumulates the encoded datafrom the reversible encoding section 106, and the processing advances tostep S46. The encoded data accumulated in the accumulation buffer 107are suitably read out and transmitted.

At step S46, the rate controlling section 117 controls the rate of thequantization operation of the quantization section 105 on the basis ofthe code amount (generated code amount) of the encoded data accumulatedin the accumulation buffer 107 such that an overflow or an underflow maynot occur, and then the encoding process is ended.

FIG. 19 is a flow chart illustrating an example of the classificationadaptive process performed at step S42 of FIG. 18.

In the image conversion apparatus 133 (FIG. 15) of the classificationadaptive filter 111, the tap selection section 21 selects, at step S51,one of pixels that have not been determined as a notice pixel as yetfrom among pixels (of a block as) the decoding in-progress imagesupplied from the arithmetic operation section 110 as a noticed pixel(processing target pixel), and the processing advances to step S52.

At step S52, the tap selection sections 21 and 22 select pixels to bemade prediction taps and class taps regarding the noticed pixel fromwithin the decoding in-progress image supplied from the arithmeticoperation section 110, respectively. Then, the tap selection section 21supplies the prediction tap to the prediction arithmetic operationsection 25, and the tap selection section 22 supplies the class tap tothe classification section 23.

Thereafter, the processing advances from step S52 to step S53, at whichthe classification section 23 performs classification of the noticedpixel using the class tap regarding the noticed pixel and the encodinginformation regarding the noticed pixel.

In particular, in the classification, the classification section 23extracts (calculates), at step S61, an image characteristic amount suchas, for example, an ADRC code (ADRC value) from the pixels thatconfigure the class tap from the tap selection section 22. Thereafter,the processing advances to step S62.

At step S62, the classification section 23 acquires necessary encodinginformation regarding the noticed pixel and converts the encodinginformation into an information code in accordance with a ruledetermined in advance. Then, the processing advances to step S63.

In particular, in the case where the encoding information is, forexample, a picture type such as an I picture, a P picture or a Bpicture, since, for example, information codes 0, 1 and 2 are allocatedto an I picture, a P picture and a B picture, respectively, the picturetype of the noticed pixel is converted into an information code inaccordance with the allocation.

At step S63, the classification section 23 generates a class coderepresentative of the class of the noticed pixel from the imagecharacteristic amount and the information code and supplies the classcode to the coefficient acquisition section 151, and the classificationat step S53 is ended.

For example, in the case where the image characteristic amount is anADRC code, the classification section 23 generates a numerical valueobtained by adding the information code to upper bits of the ADRC codeas the information characteristic amount as a class code representativeof the class of the noticed pixel.

After the classification at step S53 ends, the processing advances tostep S54, at which the coefficient acquisition section 151 acquires tapcoefficients of the class represented by the class code supplied fromthe classification section 23 from among the tap coefficients for theindividual classes stored in the storage section 162 (FIG. 16) andsupplies the tap coefficients to the prediction arithmetic operationsection 25. Then, the processing advances to step S55.

Here, the tap coefficients for the individual classes stored in thestorage section 162 of the coefficient acquisition section 151 (FIG. 16)are updated using selection coefficients as reduction filter informationfrom the reduction apparatus 132 at step S20 of FIG. 17.

At step S55, the prediction arithmetic operation section 25 performs aprediction arithmetic operation of the expression (1) using theprediction tap from the tap selection section 21 and the tapcoefficients from the coefficient acquisition section 151. Consequently,the prediction arithmetic operation section 25 determines the predictionvalue of the pixel value of the corresponding pixel of the originalimage corresponding to the noticed pixel as a pixel value of theafter-filter image, and the processing advances to step S56.

At step S56, the tap selection section 21 decides whether a pixel thathas not been determined as a noticed pixel as yet remains in the pixelsof (the block as the) decoding in-progress image from the arithmeticoperation section 110. In the case where it is decides at step S56 thata pixel that has not been determined as a noticed pixel as yet remains,the processing returns to step S51, and similar processes are repeatedthereafter.

On the other hand, in the case it is decided at step S56 that a pixelthat has not been determined as a noticed pixel as yet does not remain,the processing advances to step S57, at which the prediction arithmeticoperation section 25 supplies the after-filter image configured from thepixel values obtained for (the block as the) decoding in-progress imagefrom the arithmetic operation section 110 to the frame memory 112 (FIG.9). Then, the classification adaptive process is ended, and theprocessing returns.

As described above, in the encoding apparatus 11 of FIG. 9, since notthe latest coefficients of all classes but the latest coefficients ofthe classes selected as selection coefficients are transmitted asreduction filter information, the compression efficiency can be improvedin comparison with that in an alternative case in which the latestcoefficients of all classes are transmitted.

Here, by setting the class number to be used for a classificationadaptive process to a great number, basically it is possible to improvethe S/N of the decoded image. However, where the class number is set toa great number, if tap coefficients of such a class number as justdescribed are transmitted from the encoding apparatus 11 to the decodingapparatus 12, the compression efficiency decreases.

Therefore, the encoding apparatus 11 selects tap coefficients of classesthat are superior in merit decision value (in the present embodiment,equal to or higher than a threshold value) from among coefficients asthe latest coefficients for the individual classes as selectioncoefficients and transmits not the tap coefficients of all classes butonly the selection coefficients. Consequently, the S/N of the decodedimage can be improved and the compression efficiency can be improved.

Here, in regard to a class of the latest coefficients that is notsuperior in merit decision value (hereinafter referred to also asnon-update class), for example, the latest coefficients and thecoefficients at present have similar values to each other. As a case inwhich the latest coefficients and the coefficients at present havesimilar values to each other, a case is available in which, for example,the original image (and decoding in-progress image) that has been usedfor tap coefficient learning of the latest coefficients and the originalimage (and decoding in-progress image) that has been used for tackcoefficient learning of the coefficients at present are images havingidentity.

In particular, in regard to a non-update class, a series of originalimages from an original image in the past in time used for the tapcoefficient learning of the current coefficient to an original imagelatest in time used for the tap coefficient learning of the latestcoefficients have identity in the time direction, and the latestcoefficients and the coefficients at present come to have similar valuesto each other arising from the identity in the time direction.

Then, if the latest coefficients have values similar to those of thecoefficients at present, then whichever one of the latest coefficientsand the coefficients at present is used in the classification adaptiveprocess, there is little influence on the S/N of the decoded image.

Therefore, the encoding apparatus 11 does not transmit the latestcoefficients of non-update classes that have little influence on the S/Nof the decoded image such that the compression efficiency is improved.Accordingly, it can be considered that such improvement of thecompression rate is improvement of the compression rate that utilizesthe identity of original images (tap coefficients) in the timedirection.

<First Configuration Example of Decoding Apparatus 12>

FIG. 20 is a block diagram depicting a first configuration example ofthe decoding apparatus 12 of FIG. 1.

Referring to FIG. 20, the decoding apparatus 12 includes an accumulationbuffer 201, a reversible decoding section 202, a dequantization section203, an inverse orthogonal transform section 204, an arithmeticoperation section 205, a classification adaptive filter 206, a sortingbuffer 207 and a D/A conversion section 208. The decoding apparatus 12further includes a frame memory 210, a selection section 211, anintra-prediction section 212, a motion prediction compensation section213 and a selection section 214.

The accumulation buffer 201 temporarily accumulates encoded datatransmitted from the encoding apparatus 11 and supplies the encoded datato the reversible decoding section 202 at a predetermined timing.

The reversible decoding section 202 acquires the encoded data from theaccumulation buffer 201.

Accordingly, the reversible decoding section 202 functions as anacceptance section that accepts encoded data transmitted from theencoding apparatus 11 and eventually accepts encoding information andreduction filter information included in the encoded data.

The reversible decoding section 202 decodes the encoded data acquiredfrom the accumulation buffer 201 by a method corresponding to theencoding method of the reversible encoding section 106 of FIG. 9.

Then, the reversible decoding section 202 supplies quantizationcoefficients obtained by decoding of the encoded data to thedequantization section 203.

Further, in the case where encoding information or reduction filterinformation is obtained by decoding of the encoded data, the reversibledecoding section 202 supplies necessary encoding information to theintra-prediction section 212, motion prediction compensation section 213and other necessary blocks.

Furthermore, the reversible decoding section 202 supplies the encodinginformation and the reduction filter information to the classificationadaptive filter 206.

The dequantization section 203 dequantizes the quantization coefficientsfrom the reversible decoding section 202 by a method corresponding tothe quantization method of the quantization section 105 of FIG. 9 andsupplies transform coefficients obtained by the dequantization to theinverse orthogonal transform section 204.

The inverse orthogonal transform section 204 inversely orthogonallytransforms the transform coefficients supplied from the dequantizationsection 203 by a method corresponding to the orthogonal transform methodof the orthogonal transform section 104 of FIG. 9 and supplies aresidual obtained as a result of the inverse orthogonal transform to thearithmetic operation section 205.

To the arithmetic operation section 205, not only the residual issupplied from the inverse orthogonal transform section 204, but also aprediction image is supplied from the intra-prediction section 212 orthe motion prediction compensation section 213 through the selectionsection 214.

The arithmetic operation section 205 adds the residual from the inverseorthogonal transform section 204 and the prediction image from theselection section 214 to generate a decoding in-progress image andsupplies the decoding in-progress image to the classification adaptivefilter 206 or to the sorting buffer 207 and the frame memory 210. Forexample, decoding in-progress images that are to be made referenceimages to be used for intra-prediction from among decoding in-progressimages are supplied to the sorting buffer 207 and the frame memory 210,and the other decoding in-progress images are supplied to theclassification adaptive filter 206.

The classification adaptive filter 206 performs a classificationadaptive process similarly to the classification adaptive filter 111 ofFIG. 9 to perform ILF processing (processing of an ILF) by aclassification adaptive process using a filter that functions as an ILF.

In particular, the classification adaptive filter 206 performs (imageconversion by) a classification adaptive process using tap coefficientsfor each class obtained using the reduction filter information from thereversible decoding section 202 using the decoding in-progress imagefrom the arithmetic operation section 205 as a first image by usingencoding information from the reversible decoding section 202 asoccasion demands to convert the decoding in-progress image as a firstimage into an after-filter image as a second image that corresponds tothe original image (to generate an after-filter image), and outputs theafter-filter image.

The after-filter image outputted from the classification adaptive filter206 is an image similar to an after-filter image outputted from theclassification adaptive filter 111 of FIG. 9, and is supplied to thesorting buffer 207 and the frame memory 210.

The sorting buffer 207 temporarily stores a decoding in-progress imagesupplied from the arithmetic operation section 205 or an after-filterimage supplied from the classification adaptive filter 206 as a decodedimage, and sorts the arrangement of frames (pictures) of the decodedimage from the encoding (decoding) order to a displaying order andoutputs the sorted decoded image to the D/A conversion section 208.

The D/A conversion section 208 D/A converts the decoded image suppliedfrom the sorting buffer 207 and outputs the resulting analog decodedimage to a display not depicted such that it is displayed on thedisplay.

The frame memory 210 temporarily stores a decoding in-progress imagesupplied from the arithmetic operation section 205 or an after-filterimage supplied from the classification adaptive filter 206 as a decodedimage. Further, the frame memory 210 supplies the decoded image as areference image to be used for generation of a prediction image to theselection section 211 at a predetermined timing on the basis of anexternal request from the intra-prediction section 212, motionprediction compensation section 213 or the like.

The selection section 211 selects a supplying destination of thereference image supplied from the frame memory 210. In the case where anintra-encoded image is to be decoded, the selection section 211 suppliesthe reference image supplied from the frame memory 210 to theintra-prediction section 212. On the other hand, in the case where aninter-encoded image is to be decoded, the selection section 211 suppliesthe reference image supplied from the frame memory 210 to the motionprediction compensation section 213.

The intra-prediction section 212 performs intra prediction using areference image supplied from the frame memory 210 through the selectionsection 211 in the intra-prediction mode used by the intra-predictionsection 114 of FIG. 9 in accordance with a prediction mode included inencoding information supplied from the reversible decoding section 202.Then, the intra-prediction section 212 supplies a prediction imageobtained by the intra prediction to the selection section 214.

The motion prediction compensation section 213 performs inter predictionusing a reference image supplied from the frame memory 210 through theselection section 211 in the intra-prediction mode used by the motionprediction compensation section 115 of FIG. 9 in accordance with aprediction mode included in encoding information supplied from thereversible decoding section 202. The inter prediction is performed usingmotion information or the like included in encoding information suppliedfrom the reversible decoding section 202 as occasion demands.

The motion prediction compensation section 213 supplies the predictionimage obtained by the inter prediction to the selection section 214.

The selection section 214 selects the prediction image supplied from theintra-prediction section 212 or the prediction image supplied from themotion prediction compensation section 213 and supplies the selectedprediction image to the arithmetic operation section 205.

<Configuration Example of Classification Adaptive Filter 206>

FIG. 21 is a block diagram depicting a configuration example of theclassification adaptive filter 206 of FIG. 20.

Referring to FIG. 21, the classification adaptive filter 206 includes animage conversion apparatus 231.

To the image conversion apparatus 231, a decoding in-progress image issupplied from the arithmetic operation section 205 (FIG. 20) andreduction filter information and encoding information are supplied fromthe reversible decoding section 202.

The image conversion apparatus 231 performs image conversion by aclassification adaptive process using tap coefficients for theindividual classes using a decoding in-progress image as a first imageto convert the decoding in-progress image as the first image into anafter-filter image as a second image equivalent to an original imagesimilarly to the image conversion apparatus 133 of FIG. 10, and suppliesthe after-filter image to the sorting buffer 207 and the frame memory210 (FIG. 20).

It is to be noted that the image conversion apparatus 231 obtains(updates) tap coefficients to be used in a classification adaptiveprocess using reduction filter information similarly to the imageconversion apparatus 133 of FIG. 10.

Further, the image conversion apparatus 231 performs, in aclassification adaptive process, classification using encodinginformation as occasion demands similarly to the image conversionapparatus 133 of FIG. 10.

<Configuration Example of Image Conversion Apparatus 231>

FIG. 22 is a block diagram depicting a configuration example of theimage conversion apparatus 231 of FIG. 21.

Referring to FIG. 22, the image conversion apparatus 231 includes tapselection sections 241 and 242, a classification section 243, acoefficient acquisition section 244 and a prediction arithmeticoperation section 245.

The components from the tap selection section 241 to the predictionarithmetic operation section 245 are configured similarly to thecomponents from the tap selection section 21 to the classificationsection 23, the coefficient acquisition section 151 and the predictionarithmetic operation section 25 that configure the image conversionapparatus 133 (FIG. 15), respectively.

To the tap selection sections 241 and 242, a decoding in-progress imageis supplied from the arithmetic operation section 205 (FIG. 20).

The tap selection section 241 uses the decoding in-progress image fromthe arithmetic operation section 205 as a first image to successivelyselect pixels of the decoding in-progress image as a noticed pixel.

Further, the tap selection section 241 selects, in regard to the noticedpixel, a prediction tap of a structure same as the structure of aprediction tap selected by the tap selection section 21 of FIG. 15 fromwithin the decoding in-progress image, and supplies the selectedprediction tap to the prediction arithmetic operation section 245.

The tap selection section 242 uses a decoding in-progress image from thearithmetic operation section 205 as a first image to select, in regardto the noticed pixel, a class tap of a structure same as the structureof a class tap selected by the tap selection section 22 of FIG. 15 fromthe pixels of the decoding in-progress image, and supplies the selectedclass tap to the classification section 243.

To the classification section 243, not only the class tap is suppliedfrom the tap selection section 242 but also encoding information issupplied from the reversible decoding section 202 (FIG. 20).

The classification section 243 uses the class tap from the tap selectionsection 242 and the encoding information from the reversible decodingsection 202 to perform classification same as that by the classificationsection 23 of FIG. 15 and supplies (a class code representative of) theclass of the noticed pixel to the coefficient acquisition section 244.

To the coefficient acquisition section 244, not only the class of thenoticed pixel is supplied from the classification section 243 but alsoreduction filter information is supplied from the reversible decodingsection 202.

The coefficient acquisition section 244 uses a selection coefficient asthe reduction filter information from the reversible decoding section202 to obtain tap coefficients for each class to be used in theclassification adaptive process. Then, the coefficient acquisitionsection 244 acquires the tap coefficients of the class of the noticedpixel from the classification section 243 from among the tapcoefficients for the individual classes obtained using the selectioncoefficients as the reduction filter information and supplies theacquired tap coefficients to the prediction arithmetic operation section245.

The prediction arithmetic operation section 245 performs predictionarithmetic operation of the expression (1) using the prediction tap fromthe tap selection section 241 and the tap coefficients from thecoefficient acquisition section 244 to determine a prediction value ofthe pixel value of a corresponding pixel of the original imagecorresponding to the noticed pixel of the decoding in-progress image asa pixel value of the pixel of the decoding in-progress image as a secondimage, and outputs the determined pixel value of the pixel.

FIG. 23 is a block diagram depicting a configuration example of thecoefficient acquisition section 244 of FIG. 22.

Referring to FIG. 23, the coefficient acquisition section 244 includesan updating section 251, a storage section 252 and an acquisitionsection 253.

To the updating section 251, selection coefficients as reduction filterinformation are supplied from the reversible decoding section 202 (FIG.20).

The updating section 251 updates tap coefficients for the individualclasses as coefficients at present stored in the storage section 252with the selection coefficients as reduction filter information from thereversible decoding section 202.

The storage section 252 stores tap coefficients for the individualclasses.

Here, as described hereinabove with reference to FIG. 16, in theencoding apparatus 11, at an initialization timing such as, for example,a timing at which the power supply is turned on or a timing at whichencoding of a sequence of an original image is started, the reductionapparatus 132 (FIG. 10) selects the tap coefficients of all classes asthe latest coefficients from the learning apparatus 131 as selectioncoefficients and outputs the tap coefficients as reduction filterinformation to the reversible encoding section 106.

Accordingly, at the initialization timing, reduction filter informationtransmitted from the encoding apparatus 11 to the decoding apparatus 12is tap coefficients (latest coefficients) of all classes, and theupdating section 251 stores the tap coefficients of all classes as thereduction filter information into the storage section 252.

In the image conversion apparatus 231 (FIG. 22), a classificationadaptive process is performed using the tap coefficients for theindividual classes stored in the storage section 252 in such a manner asdescribed above as the coefficients at present.

Then, when an update timing comes thereafter and reduction filterinformation is transmitted from the encoding apparatus 11 to thedecoding apparatus 12, the updating section 251 updates the currentcoefficients of the class of the selection coefficients from among thetap coefficients as the coefficients at present stored in the storagesection 252 with and to the selection coefficients as the reductionfilter information from the encoding apparatus 11.

To the acquisition section 253, (a class code of) a class of a noticedpixel is supplied from the classification section 243. The acquisitionsection 253 acquires tap coefficients as the coefficients at present ofthe class of the noticed pixel from the coefficients at present storedin the storage section 252 and supplies the tap coefficients to theprediction arithmetic operation section 245.

<Decoding Process>

FIG. 24 is a flow chart illustrating an example of a decoding process ofthe decoding apparatus 12 of FIG. 20.

It is to be noted that the order of steps of the decoding processdepicted in FIG. 24 and so forth is an order for the convenience ofdescription, and steps of an actual encoding process are performedsuitably in parallel in a necessary order. This similarly applies alsoto the decoding process hereinafter described.

In the decoding process, at step S71, the accumulation buffer 201temporarily accumulates encoded data transmitted from the encodingapparatus 11 and suitably supplies the encoded data to the reversibledecoding section 202. Then, the processing advances to step S72.

At step S72, the reversible decoding section 202 receives and decodesthe encoded data supplied from the accumulation buffer 201 and suppliesquantization coefficients obtained by the decoding to the dequantizationsection 203.

Further, in the case where encoding information or reduction filterinformation is obtained by the decoding of the encoded data, thereversible decoding section 202 supplies necessary encoding informationto the intra-prediction section 212, motion prediction compensationsection 213 and other necessary blocks.

Furthermore, the reversible decoding section 202 supplies the encodinginformation and the reduction filter information to the classificationadaptive filter 206.

Thereafter, the processing advances from step S72 to step S73, at whichthe classification adaptive filter 206 decides whether reduction filterinformation is supplied from the reversible decoding section 202.

In the case where it is decided at step S73 that reduction filterinformation is not supplied, the processing advances to step S75skipping step S74.

On the other hand, in the case where it is decided at step S73 thatreduction filter information is supplied, the processing advances tostep S74, at which the updating section 251 of the coefficientacquisition section 244 (FIG. 23) that configures the image conversionapparatus 231 (FIG. 22) of the classification adaptive filter 206acquires selection coefficients as the reduction filter information fromthe reversible decoding section 202 and updates the tap coefficients forthe individual classes as the coefficients at present stored in thestorage section 252 with the selection coefficients as the reductionfilter information.

Then, the processing advances from step S74 to step S75, at which aprediction decoding process is performed, and then the decoding processcomes to an end.

FIG. 25 is a flow chart illustrating an example of the predictiondecoding process at step S75 of FIG. 24.

At step S81, the dequantization section 203 dequantizes the quantizationcoefficients from the reversible decoding section 202 and suppliesconversion coefficients obtained as a result of the dequantization tothe inverse orthogonal transform section 204. Then, the processingadvances to step S82.

At step S82, the inverse orthogonal transform section 204 inverselyorthogonally transforms the conversion coefficients from thedequantization section 203 and supplies a residual obtained as a resultof the inverse orthogonal transform to the arithmetic operation section205, and then the processing advances to step S83.

At step S83, the intra-prediction section 212 or the motion predictioncompensation section 213 performs a prediction process for generating aprediction image using a reference image supplied from the frame memory210 through the selection section 211 and encoding information suppliedfrom the reversible decoding section 202. Then, the intra-predictionsection 212 or the motion prediction compensation section 213 supplies aprediction image obtained by the prediction process to the selectionsection 214, and the processing advances from step S83 to step S84.

At step S84, the selection section 214 selects the prediction imagesupplied from the intra-prediction section 212 or the motion predictioncompensation section 213 and supplies the selected prediction image tothe arithmetic operation section 205. Then, the processing advances tostep S85.

At step S85, the arithmetic operation section 205 adds the residual fromthe inverse orthogonal transform section 204 and the prediction imagesupplied from the selection section 214 to generate a decodingin-progress image. Then, the arithmetic operation section 205 suppliesthe decoding in-progress image to the classification adaptive filter 206or the sorting buffer 207 and the frame memory 210, and then theprocessing advances from step S85 to step S86.

In the case where the decoding in-progress image is supplied from thearithmetic operation section 205 to the classification adaptive filter206, the classification adaptive filter 206 performs, at step S86, aclassification adaptive process (classification adaptive filter process)as a process of an ILF for the decoding in-progress image from thearithmetic operation section 205. By performing the classificationadaptive process for the decoding in-progress image, similarly as in thecase of the encoding apparatus 11, an after-filter image closer to theoriginal image than that in the case where the decoding in-progressimage is filtered by an ILF is determined.

The classification adaptive filter 206 supplies the decoding in-progressimage obtained by the classification adaptive process to the sortingbuffer 207 and the frame memory 210, and the processing advances fromstep S86 to step S87.

At step S87, the sorting buffer 207 temporarily stores the decodingin-progress image supplied from the arithmetic operation section 205 orthe decoding in-progress image supplied from the classification adaptivefilter 206 as a decoded image. Further, the sorting buffer 207 sorts thestored decoded image into a displaying order and supplies the sorteddecoded image to the D/A conversion section 208, and the processingadvances from step S87 to step S88.

At step S88, the D/A conversion section 208 D/A converts the decodedimage from the sorting buffer 207, and the processing advances to stepS89. The decoded image after the D/A conversion is outputted to anddisplayed on the display not depicted.

At step S89, the frame memory 210 temporarily stores the decodingin-progress image supplied from the arithmetic operation section 205 orthe decoding in-progress image supplied from the classification adaptivefilter 206 as a decoded image, and the decoding process comes to an end.The decoded image stored in the frame memory 210 is used as a referenceimage that is made an original from which a prediction image is to begenerated by the prediction process at step S83.

FIG. 26 is a flow chart illustrating an example of the classificationadaptive process performed at step S86 of FIG. 24.

In the image conversion apparatus 231 (FIG. 22) of the classificationadaptive filter 206, at step S91, the tap selection section 241 selectsone of pixels that have not been set as a noticed pixel as yet fromamong the pixels (of a block as) the decoding in-progress image suppliedfrom the arithmetic operation section 205 as a noticed pixel, and theprocessing advances to step S92.

At step S92, the tap selection sections 241 and 242 select pixels to bemade prediction taps and class taps in regard to the noticed pixel fromwithin the decoding in-progress image supplied from the arithmeticoperation section 205, respectively. Then, the tap selection section 241supplies the prediction taps to the prediction arithmetic operationsection 245, and the tap selection section 242 supplies the class tapsto the classification section 243.

Thereafter, the processing advances from step S92 to step S93, at whichthe classification section 243 performs classification of the noticedpixel similar to that in the case described hereinabove with referenceto FIG. 19 using the class taps regarding the noticed pixel suppliedfrom the tap selection section 242 and the encoding informationregarding the noticed pixel supplied from the reversible decodingsection 202.

The classification section 243 generates a class code representative ofthe class of the noticed pixel obtained by the classification andsupplies the generated class code to the coefficient acquisition section244, and the processing advances from step S93 to step S94.

At step S94, the coefficient acquisition section 244 acquires tapcoefficients of the class represented by the class code supplied fromthe classification section 243 from the tap coefficients stored in thestorage section 252 (FIG. 23) updated at step S74 of FIG. 24 andsupplies the acquired tap coefficients to the prediction arithmeticoperation section 245, and the processing advances to step S95.

At step S95, the prediction arithmetic operation section 245 performsprediction arithmetic operation of the expression (1) using theprediction taps from the tap selection section 241 and the tapcoefficients from the coefficient acquisition section 244. Consequently,the prediction arithmetic operation section 245 determines theprediction value of the pixel value of the corresponding pixel of theoriginal image corresponding to the noticed pixel as a pixel value ofthe after-filter image, and the processing advances to step S96.

At step S96, the tap selection section 241 decides whether a pixel thathas not been set as a notice pixel as yet exists in the pixels (of ablock as) the decoding in-progress image from the arithmetic operationsection 205. In the case where it is decided at step S96 that a pixelthat has not been set as a noticed pixel as yet exits, the processingreturns to step S91 and similar processes are repeated thereafter.

On the other hand, in the case where it is decided at step S96 that apixel that has not been set as a noticed pixel does not exit, theprocessing advances to step S97, at which the prediction arithmeticoperation section 245 supplies an after-filter image configured frompixel values obtained for (a block as) the decoding in-progress imagefrom the arithmetic operation section 205 to the sorting buffer 207 andthe frame memory 210 (FIG. 9). Then, the classification adaptive processis encoded, and the processing returns.

As described above, in the encoding apparatus 11 of FIG. 9 and thedecoding apparatus 12 of FIG. 20, since an ILF process is performed by aclassification adaptive process, an after-filter image closer to theoriginal image than that by a processing result of the ILF can beobtained. As a result, the S/N of the decoded image can be improved.Further, since an after-filter image close to the original image can beobtained, the residual reduces, and therefore, the compressionefficiency can be improved. Further, in the encoding apparatus 11, sincethe latest coefficients of classes in regard to which the limit decisionvalue is equal to or higher than a threshold value to generate reductionfilter information that decreases the tap coefficients for theindividual classes and not the tap coefficients of the latestcoefficients of all classes but the reduction filter information istransmitted to the decoding apparatus 12, the compression efficiency canbe improved further.

It is to be noted that, as the ILF, for example, a DF (DeblockingFilter) for reducing block noise, a SAO (Sample Adaptive Offset) forreducing ringing and an ALF (Adaptive Loop Filter) for minimizing theencoding error (error of the decoded image with respect to the originalimage) are available.

The DF controls the filter strength depending upon the quantizationparameter QP or upon whether or not the pixel is a pixel on the boundaryof a block to reduce block noise (distortion).

However, in the DF, the number of filter strengths that can be appliedto a block is as small as two. Further, in the DF, the unit of controlof the filter strength is slice, and the filter strength cannot becontrolled for each pixel.

In the SAO, the filter mode in which noise around an edge is to bereduced or DC correction is to be performed is changed over for eachCTU, and by deciding an offset value for each pixel, reduction ofringing or DC correction is performed.

However, in the SAO, it is difficult to perform changeover of the filtermode for each pixel. Further, in the SAO, only one of the processes ofreduction of noise and DC correction can be performed, and bothprocesses cannot be performed simultaneously.

The ALF performs classification for classifying into 15 classes usingthe direction of an edge and the activity and performs a filter processbased on filter coefficients prepared statistically optimally.

However, in the ALF, since the unit of a filter process is a unit of 4×4pixels, for each pixel, fine control of the filter strength according toa waveform pattern or a block phase around the pixel cannot beperformed. Further, in the ALF, since the class number that becomes atarget of classification is as small as 15 classes, fine control of thefilter strength cannot be performed also from this point.

In contrast, in the classification adaptive process, since, for eachpixel, classification is performed for a target of a class numbergreater than 15 classes of the ALF and tap coefficients that areobtained by learning and are statistically optimum are used to perform afilter process for converting a decoding in-progress image into anafter-filter image, the picture quality (S/N) can be improvedsignificantly from that of the existing ILF.

Especially, in the classification adaptive process, since, for eachpixel, classification is performed using an image characteristic amountsuch as an ADRC code, a dynamic range or the like as an imagecharacteristic amount that represents a waveform pattern around thepixel and encoding information such as a quantization parameter QP, apicture type, a block phase or the like, an image very close to theoriginal image can be obtained as an after-filter image. As a result,not only in the case where the ILF is not used in prediction encoding,but also in comparison with an alternative case in which the ILF isused, the S/N and the compression efficiency of images can be improvedsignificantly.

Further, the classification adaptive process does not have, in regard totap coefficients to be used for prediction arithmetic operation of theexpression (1) as a filter process, a restriction of point symmetry asin the case of the ALF or such restriction that the number of filtercoefficients is 13. Therefore, tap coefficients that make a statisticalerror of the after-filter image from the original image smaller thanthat by the ALF can be determined by learning.

Furthermore, in the classification adaptive process, the number of tapcoefficients, namely, the number of pixels for configuring a predictiontap, or a structure of the prediction tap can be designed, for example,taking the data amount of tap coefficients and the S/N and thecompression efficiency of a decoded image into consideration.

Further, in the classification adaptive process, prediction taps can beconfigured including not only a frame of a noticed pixel but also pixelsof preceding and succeeding frames of the frame.

As described above, since, in the classification adaptive process,prediction arithmetic operation of the expression (1) as classificationand a filter process is performed for each pixel, it is possible tocause, for each pixel, an effect of a filter process suitable for thepixel.

As a result, for example, it is possible to cause an effect of NR (NoiseReduction) or suppress ringing without excessively crushing an edge or atexture. In particular, for example, it is possible to maintain, inregard to pixels at an edge portion, the edge portion (to leave details)and to remove, in regard to pixels at a flat portion, noisesufficiently.

Further, in the classification adaptive process, for example, byperforming classification using encoding information of the block phaseor the like, the effect of NR can be adjusted depending upon whether thenoticed pixel is a pixel on the boundary of a block. As a result, it ispossible to perform, for a pixel that suffers from block distortion, afilter process suitable to remove the block distortion and perform, fora pixel that suffers from noise other than block distortion, a filterprocess suitable to remove the noise.

Furthermore, in the classification adaptive process, by configuringprediction taps including, for example, not only a frame of the noticedpixel but also pixels of preceding and succeeding frames of the frame,degradation of the picture quality arising from a movement such asmotion blur can be moderated.

Here, in the present technology, since the classification adaptiveprocess is performed for a decoding in-progress image that is handled inthe encoding apparatus 11 and the decoding apparatus 12, a block phaseand information regarding a block necessary to specify the block phase(for example, a size of the block, a boundary of the block and so forth)can be obtained by an encoding process or a decoding process.

Meanwhile, for example, Japanese Patent No. 4770711 describes atechnology for improving the picture quality of a decoded imageoutputted from an MPEG decoder by a classification adaptive process thatuses a block phase. In the technology described in Japanese Patent No.4770711, since a classification adaptive process for a decoded imageoutputted from an MPEG decoder is performed outside the MPEG decoder, itis necessary for information regarding a block to be definitely decidedor be detected by some method.

It is to be noted that, while, in the first configuration example of theencoding apparatus 11 of FIG. 9, the ILF, namely, all processes of theDF, SAO and ALF, are performed by the classification adaptive process,in the classification adaptive process, not all of the ILF but one ortwo of the DF, SAO and ALF can be performed. This similarly applies alsoto the first configuration example of the decoding apparatus 12 of FIG.20 and other configuration examples of the encoding apparatus 11 and thedecoding apparatus 12 hereinafter described.

<Second Configuration Example of Encoding Apparatus 11>

FIG. 27 is a block diagram depicting a second configuration example ofthe encoding apparatus 11 of FIG. 1.

It is to be noted that, in FIG. 27, elements corresponding to those ofFIG. 9 are denoted by like reference numerals and description of them ishereinafter omitted suitably.

Referring to FIG. 27, the encoding apparatus 11 includes the componentsfrom the A/D conversion section 101 to the arithmetic operation section110 and from the frame memory 112 to the rate controlling section 117,and a classification adaptive filter 311.

Accordingly, the encoding apparatus 11 of FIG. 27 is common to that ofFIG. 9 in that it includes the components from the A/D conversionsection 101 to the arithmetic operation section 110 and from the framememory 112 to the rate controlling section 117.

However, the encoding apparatus 11 of FIG. 27 is different from that ofFIG. 9 in that it includes the classification adaptive filter 311 inplace of the classification adaptive filter 111.

The classification adaptive filter 311 is common to the classificationadaptive filter 111 in that it is a filter that functions as an ILF byperforming a classification adaptive process and performs an ILF processby the classification adaptive process.

However, the classification adaptive filter 311 is different from theclassification adaptive filter 111 in that tap coefficients forindividual integration classes where a plurality of classes determinedby tap coefficient learning are integrated to the number of classesequal to or smaller than the number of the plurality of classes byreduction of the tap coefficients of the classes by a reduction processand a corresponding relationship LUT (Look Up Table) as correspondingrelationship information representative of a corresponding relationshipbetween the plurality of original classes and the integration classesare generated as reduction filter information.

Further, the classification adaptive filter 311 is different from theclassification adaptive filter 111 in that it performs a classificationadaptive process using the tap coefficients for the individualintegration classes as the reduction filter information and thecorresponding relationship LUT.

<Configuration Example of Classification Adaptive Filter 311>

FIG. 28 is a block diagram depicting a configuration example of theclassification adaptive filter 311 of FIG. 27.

It is to be noted that, in FIG. 28, elements corresponding to those ofthe classification adaptive filter 111 of FIG. 10 are denoted by likereference numerals and description of them is hereinafter omittedsuitably.

Referring to FIG. 28, the classification adaptive filter 311 includes alearning apparatus 131, a reduction apparatus (reduction section) 321and an image conversion apparatus 322.

Accordingly, the classification adaptive filter 311 is common to theclassification adaptive filter 111 of FIG. 10 in that it includes thelearning apparatus 131.

However, the classification adaptive filter 311 is different from theclassification adaptive filter 111 in that it includes the reductionapparatus 321 and the image conversion apparatus 322 in place of thereduction apparatus 132 and the image conversion apparatus 133,respectively.

To the reduction apparatus 321, tap coefficients of a plurality ofclasses obtained by tap coefficient learning are supplied from thelearning apparatus 131.

Here, a class with regard to which tap coefficients are determined bythe learning apparatus 131 is referred to also as initial class.Further, it is assumed that the number of initial classes (total classnumber) is C (classes).

The reduction apparatus 321 performs a reduction process for reducingthe tap coefficients of the C initial classes from the learningapparatus 131 to generate reduction filter information and supplies thereduction filter information obtained by the reduction process to theimage conversion apparatus 322 and the reversible encoding section 106(FIG. 27).

In particular, in the reduction process, the reduction apparatus 321integrates the tap coefficients for the C individual initial classes toU classes smaller than the C classes to generate tap coefficients forthe individual integrated classes.

Further, the reduction apparatus 321 generates a correspondingrelationship LUT as corresponding relationship informationrepresentative of a corresponding relationship between the initialclasses and the integrated classes.

Then, the reduction apparatus 321 supplies the tap coefficients for theU individual integrated classes and the corresponding relationship LUTas reduction filter information to the image conversion apparatus 322and the reversible encoding section 106 (FIG. 27).

To the image conversion apparatus 322, a decoding in-progress image issupplied from the arithmetic operation section 110 (FIG. 27) andreduction filter information is supplied from the reduction apparatus321. Further, encoding information is supplied to the image conversionapparatus 322.

The image conversion apparatus 322 performs, using the decodingin-progress image as a first image, image conversion by a classificationadaptive process using the tap coefficients for the individualintegrated classes (tap coefficients obtained using the reduction filterinformation) as the reduction filter information from the reductionapparatus 321 and the corresponding relationship LUT to convert thedecoding in-progress image as the first image into an after-filter imageas the second image equivalent to the original image (generates anafter-filter image) and supplies the after-filter image to the framememory 112 (FIG. 27).

It is to be noted that the image conversion apparatus 322 performsclassification in the classification adaptive process using encodinginformation as occasion demands.

<Example of Reduction Process of Reduction Apparatus 321>

FIG. 29 is a view illustrating an example of the reduction process ofthe reduction apparatus 321.

In FIG. 29, at an update timing for tap coefficients to be used in aclassification adaptive process, tap coefficients of C initial classes0, 1, . . . , C−1 are determined by tap coefficient learning by thelearning apparatus 131.

The reduction apparatus 321 calculates a tap coefficient evaluationvalue representative of appropriateness in use of tap coefficients foreach integrated class in the case where each two or more initial classesfrom among the C initial classes 0 to C−1 are integrated into anintegration candidate class in (prediction arithmetic operation of theexpression (1) of) a classification adaptive process.

Further, the reduction apparatus 321 integrates integration candidateclasses according to the tap coefficient evaluation values and repeatssimilar integration using the tap coefficient for each class after theintegration as a target. Then, the reduction apparatus 321 outputs tapcoefficients of the U classes (integration classes) 0, 1, . . . , U−1equal to or smaller than C as reduction filter information finallyobtained by the integration.

Further, the reduction apparatus 321 generates and outputs acorresponding relationship LUT between the initial classes and theintegration classes as reduction filter information.

In FIG. 29, for example, the initial class 0 is not integrated withanother class and is set as it is as integrated class 0. Meanwhile, forexample, the initial classes 1 and 2 are integrated into an integratedclass 1. Therefore, in the corresponding relationship LUT of FIG. 29,the initial class 0 and the integrated class 0 are associated with eachother, and the initial classes 1 and 2 and the integrated claim 1 areassociated with each other.

In the classification adaptive process performed by the image conversionapparatus 322 (FIG. 28), an initial class as a class of a noticed pixelobtained by the classification is converted into an integrated class inaccordance with the corresponding relationship LUT as reduction filterinformation. Then, from the tap coefficients for the individualintegrated classes as the reduction filter information, the tapcoefficients of the integrated class of the noticed pixel are acquired,and prediction arithmetic operation is performed using the tapcoefficients.

Here, as the tap coefficient evaluation value, for example, a valuecorresponding to the RD cost can be adopted. For example, it is possibleto adopt the RD cost as the tap coefficient evaluation value and selectintegration candidate classes from initial classes to performintegration such that the RD cost is improved. Further, integration ofclasses can be performed repetitively, for example, until the RD costdoes not improve anymore (is continued as long as the RD cost improves).

By adopting the RD cost as the tap coefficient evaluation value andrepeating selection and integration of integration candidate classes inresponse to the RD cost in such a manner as described above, it ispossible to achieve optimization of classes for which integration is tobe performed and optimization of the quantity U of integration classesobtained finally by the integration and thus improve the compressionefficiency and the S/N of decoded images.

Further, as the tap coefficient evaluation value, for example, a valuecorresponding to an inter-coefficient distance between tap coefficientsof different classes can be adopted. For example, it is possible toadopt the inter-coefficient distance between tap coefficients ofdifferent classes as the tap coefficient evaluation value and repeatselection and integration of integration candidate classes such thatclasses having a small inter-coefficient distance do not remain.

Furthermore, as the tap coefficient evaluation value, for example, avalue corresponding to the S/N of an after-filter image can be adopted.For example, it is possible to adopt the S/N of an after-filter image asthe tap coefficient evaluation value and repeat selection andintegration of integration candidate classes such that the S/N of theafter-filter image is improved.

Further, as the tap coefficient evaluation value, for example, a valuecorresponding to the use frequency in which tap coefficients are used inthe classification adaptive process can be adopted. For example, it ispossible to adopt the use frequency of tap coefficients as the tapcoefficient evaluation value and repeat selection and integration ofintegration candidate classes such that classes of tap coefficientswhose use frequency, namely, classes from which the number of classesobtained as a result of classification of a noticed pixel is small, areintegrated such that integration classes with regard to which the usefrequency of tap coefficients is low do not remain.

Furthermore, as the tap coefficient evaluation value, for example, avalue corresponding to a difference between tap coefficients of a monoclass that is a specific one class and tap coefficients of a differentclass can be adopted.

Here, it is possible to adopt, as tap coefficients of a certain oneclass among initial classes, an average tap coefficient of tapcoefficients of a different class (for example, tap coefficientsobtained by tap coefficient learning setting the class number to oneclass). Now, it is assumed that this one class is referred to as monoclass.

As the tap coefficient evaluation value, for example, a differenceabsolute value sum or the like of tap coefficients of a mono class andtap coefficients of a different class can be adopted. In this case,selection and integration of integration candidate classes can berepeated such that classes of tap coefficients with regard to which thedifference absolute value sum with the tap coefficients of the monoclass is small do not remain anymore.

From the point of view of improvement of the compression efficiency andimprovement of the S/N of the decoded image, it is desirable to adopt avalue corresponding to the RD cost as the tap coefficient evaluationvalue. However, in the case where the RD cost is adopted as the tapcoefficient evaluation value, a high calculation cost is required forcalculation of the tap coefficient evaluation value. By adopting, as thetap coefficient evaluation value, for example, the inter-coefficientdistance described hereinabove, the S/N of the after-filter image or avalue corresponding to the difference from the tap coefficients of themono class, the calculation cost required for calculation of the tapcoefficient evaluation value can be reduced.

<Configuration Example of Reduction Apparatus 321>

FIG. 30 is a block diagram depicting a configuration example of thereduction apparatus 321 of FIG. 28.

Referring to FIG. 30, the reduction apparatus 321 includes a classintegration section 331, a storage section 332, a correspondingrelationship detection section 333 and another storage section 334.

To the class integration section 331, tap coefficients for each initialclass are supplied from the learning apparatus 131.

The class integration section 331 registers the tap coefficients foreach initial class from the learning apparatus 131 into a coefficientlist stored in the storage section 332 (list into which tap coefficientsfor each class are to be registered).

Further, the class integration section 331 selects, for example, twoclasses from among the classes whose tap coefficient is registered inthe coefficient list of the storage section 332 as integration candidateclasses.

Then, the class integration section 331 calculates the RD costs or thelike as the tap coefficient evaluation value before and afterintegration of the integration candidate classes, and in the case wherethe tap coefficient evaluation value after integration improves fromthat before integration, the class integration section 331 integratesthe tap coefficients of the integration candidate classes and updatesthe coefficient list of the storage section 332 with the tap coefficientof the class after the integration (for example, rewrites the tapcoefficients of the integration candidate classes into the tapcoefficients after the integration and collected).

Here, in the integration of the tap coefficients of the two classes C1and C2, the tap coefficients of one of the classes C1 and C2, an averagevalue of the tap coefficients of the classes C1 and C2 or the like canbe adopted as the tap coefficients of the class after the integration.

Further, tap coefficients of a class after integration where the classesC1 and C2 are integrated can be determined, for example, by retainingthe components (Σx_(n,k)x_(n′,k)) of the matrix of the left side and thecomponents (Σx_(n,k)y_(k)) of the vector of the right side of theexpression (8) used when the tap coefficients of the classes C1 and C2are determined, adding the components (Σ_(n,k)x_(n′,k)) or thecomponents (Σx_(n,k)y_(k)) of the vector used when the tap coefficientsof the class C1 are determined and the components (Σx_(n,k)x_(n′,k)) ofthe matrix or the components (Σx_(n,k)y_(k)) of the vector used when thetap coefficients of the class C2 are determined to determine thecomponents (Σx_(n,k)x_(n′,k)) of the matrix and the components(Σx_(n,k)y_(k)) of the vector of the class after the integration,respectively, and solving the normal equation indicated by theexpression (8) defined by the components (Σx_(n,k)x_(n′,k)) of thematrix and the components (Σx_(n,k)y_(k)) of the vector of the classafter the integration.

The class integration section 331 integrates the tap coefficients of theintegration candidate classes and updates the coefficient list of thestorage section 332 with the tap coefficients of the class after theintegration, and then supplies integration information representative ofthe integrated classes (integration candidate classes) and the classesafter the integration to the corresponding relationship detectionsection 333.

The storage section 332 has stored therein the coefficient list in whichthe tap coefficients for the individual classes are registered.

The coefficient list of the storage section 332 is updated in responseto integration of classes by the class integration section 331. Then,after the integration of classes by the class integration section 331ends, the tap coefficients registered in the coefficient list, namely,the tap coefficients of the integration class, are outputted asreduction filter information from the reduction apparatus 321.

The corresponding relationship detection section 333 detects classesbefore integration and a class after integration from the integrationinformation supplied from the class integration section 331 and updatesthe corresponding relationship LUT of the storage section 334 such thatthe classes before the integration and the class after the integrationare associated with each other.

The storage section 334 stores the corresponding relationship LUT inwhich a corresponding relationship between the initial classes and theintegration classes is registered.

In the corresponding relationship LUT stored in the storage section 334,as initial values, for example, information that associates initialclasses c and integration classes c with each other is registered.

Then, the corresponding relationship LUT is updated by the correspondingrelationship detection section 333 such that, for example, in the casewhere the initial classes c1 and c2 are integrated and the class afterthe integration is a class u1, the initial classes c1 and c2 and theintegration class u1 are associated with each other.

<Configuration Example of Image Conversion Apparatus 322>

FIG. 31 is a block diagram depicting a configuration example of theimage conversion apparatus 322 of FIG. 28.

It is to be noted that, in FIG. 31, elements corresponding to those ofthe image conversion apparatus 133 of FIG. 15 are denoted by likereference numerals and description of them is hereinafter omittedsuitably.

Referring to FIG. 31, the image conversion apparatus 322 includes thecomponents from the tap selection section 21 to the classificationsection 23, the prediction arithmetic operation section 25 and acoefficient acquisition section 341.

Accordingly, the image conversion apparatus 322 is configured similarlyto the image conversion apparatus 133 of FIG. 15 in that it includes thecomponents from the tap selection section 21 to the classificationsection 23 and the prediction arithmetic operation section 25.

However, the image conversion apparatus 322 is different from the imageconversion apparatus 133 in that it includes the coefficient acquisitionsection 341 in place of the coefficient acquisition section 151.

To the coefficient acquisition section 341, tap coefficients for eachintegration class as reduction filter information and a correspondingrelationship LUT are supplied from the reduction apparatus 321.

The coefficient acquisition section 341 determines tap coefficients forindividual integration classes as reduction filter information from thereduction apparatus 321 as tap coefficients for individual classes to beused for a classification adaptive process and acquires the tapcoefficient of the integration class of a noticed pixel from among thetap coefficients for the individual integration classes, and thensupplies the tap coefficient to the prediction arithmetic operationsection 25.

In particular, the coefficient acquisition section 341 converts theclass (initial class) of the noticed pixel from the classificationsection 23 into an integration class in accordance with thecorresponding relationship LUT as the reduction filter information fromthe reduction apparatus 321. Further, the coefficient acquisitionsection 341 acquires the tap coefficient of the integrated class of thenoticed pixel from among the tap coefficients for the individual classesas the reduction filter information, and then supplies the tapcoefficient to the prediction arithmetic operation section 25.

FIG. 32 is a block diagram depicting a configuration example of thecoefficient acquisition section 341 of FIG. 31.

Referring to FIG. 32, the coefficient acquisition section 341 includes astorage section 351, an integration class conversion section 352 and anacquisition section 353.

To the storage section 351, tap coefficients for each integration classas reduction filter information are supplied from the reductionapparatus 321 (FIG. 30).

The storage section 351 has stored therein tap coefficients for theindividual integration classes as the reduction filter information fromthe reduction apparatus 321.

To the integration class conversion section 352, an initial class as aclassification result of a noticed pixel is supplied from theclassification section 23. Further, to the integration class conversionsection 352, a corresponding relationship LUT as reduction filterinformation is supplied from the reduction apparatus 321.

The integration class conversion section 352 converts the initial classof the noticed pixel into an integration class of the noticed pixel inaccordance with the corresponding relationship LUT and supplies theintegration class to the acquisition section 353.

The acquisition section 353 acquires the tap coefficient of theintegration class of the noticed pixel from the integration classconversion section 352 from among the tap coefficients for theindividual classes stored in the storage section 351 and supplies theacquired tap coefficient to the prediction arithmetic operation section25.

<Encoding Process>

FIG. 33 is a flow chart illustrating an example of an encoding processof the encoding apparatus 11 of FIG. 27.

At step S111, the learning apparatus 131 (FIG. 28) of the classificationadaptive filter 311 decides whether the timing at present is an updatetiming for tap coefficients similarly as at step S11 of FIG. 17.

In the case where it is decided at step S111 that the timing at presentis not an update timing for tap coefficients, the processing advances tostep S123 skipping steps S112 to S122.

On the other hand, in the case where it is decided at step S111 that thetiming at present is an update timing for tap coefficients, theprocessing advances to step S112, at which the learning apparatus 131performs tap coefficient learning similarly as at step S12 of FIG. 17.

Then, the learning apparatus 131 supplies tap coefficients for theindividual classes (initial classes) obtained by the tap coefficientlearning to the reduction apparatus 321, and the processing advancesfrom step S112 to step S113.

At step S113, the class integration section 331 of the reductionapparatus 321 (FIG. 30) registers the tap coefficients for theindividual initial classes whose tap coefficients are supplied from thelearning apparatus 131 into the coefficient list of the storage section332. Further, the class integration section 331 calculates aninter-coefficient distance in regard to all combinations of two classesfrom among the classes whose tap coefficient is registered in thecoefficient list, and the processing advances from step S113 to stepS114.

At step S114, the class integration section 331 selects two classes of acombination in which the inter-coefficient distance is smallest fromamong the combinations of two classes from among the classes whose tapcoefficient is registered in the coefficient list of the storage section332 (FIG. 30) as integration candidate classes, and the processingadvances to step S115.

At step S115, the class integration section 331 calculates the tapcoefficients of a class after integration in the case where the twointegration candidate classes are integrated into one class, and theprocessing advances to step S116.

At step S116, the class integration section 331 calculates a tapcoefficients evaluation value such as, for example, the RD cost inregard to tap coefficients for each integration class after the twointegration candidate classes are integrated into one class at theimmediately preceding step S115. Then, the processing advances to stepS117.

Here, in the case where the step S116 is performed first after tapcoefficients for each initial class are obtained, the learning apparatus131 calculates a tap coefficient evaluation value in regard to the tapcoefficients for each integration class after two integration candidateclasses are integrated into one class, and calculates a tap coefficientevaluation value also in regard to the tap coefficients for each classbefore the integration, namely, in regard to the tap coefficients foreach initial class.

At step S117, the class integration section 331 decides whether the tapcoefficient evaluation value regarding tap coefficients for eachintegration class after two integration candidate classes are integratedinto one class is improved from the tap coefficient evaluation valuebefore the integration.

In the case where it is decided at step S117 that the tap coefficientevaluation value regarding tap coefficients for each integration classafter two integration candidate classes are integrated into one class isimproved, the processing advances to step S118.

At step S118, the class integration section 331 updates the coefficientlist of the storage section 332 (FIG. 30) with the tap coefficients forthe individual integration classes after two integration candidateclasses are integrated into one class.

Further, the class integration section 331 supplies integrationinformation representative of the two integration candidate classesintegrated with each other and the class after the integration to thecorresponding relationship detection section 333 (FIG. 30). Thecorresponding relationship detection section 333 updates thecorresponding relationship LUT of the storage section 334 such that thetwo integration candidate classes integrated with each other and theclass after the integration are associated with each other in accordancewith the integration information supplied from the class integrationsection 331, and the processing advances from step S118 to step S119.

At step S119, the class integration section 331 calculates theinter-coefficient distance in regard to all of combinations of a classafter integration (hereinafter referred to also as latest integrationclass) obtained by integration of two integration candidate classesperformed immediately before then and a different class whose tapcoefficient is registered in the coefficient list of the storage section332.

Here, since the inter-coefficient distance in regard to the combinationof two classes other than the latest integration class from among theclasses whose tap coefficient is registered in the coefficient list ofthe storage section 332 has been calculated at step S113 or S119performed in the past, later processing can be performed utilizing theinter-coefficient distance calculated already.

After the inter-coefficient distance is calculated at step S119, theprocessing returns to step S114, whereafter similar processes arerepeated.

Then, in the case where it is decided at step S117 that the tapcoefficient evaluation value in regard to the tap coefficients for eachintegration class after two integration candidate classes are integratedinto one class is not improved, namely, in the case where, even ifintegration of the class is performed, the RD cost as the tapcoefficient evaluation value does not improve any more, the processingadvances to step S120.

At step S120, the reduction apparatus 321 outputs the tap coefficientsfor the individual classes (integration classes) registered in thecoefficient list of the storage section 332 and the correspondingrelationship LUT of the storage section 334 as reduction filterinformation to the reversible encoding section 106 (FIG. 27) and theimage conversion apparatus 322 (FIG. 28), and the processing advances tostep S121.

At step S121, the reversible encoding section 106 (FIG. 27) sets thereduction filter information from the reduction apparatus 321 as atransmission target, and the processing advances to step S122. Thereduction filter information set as the transmission target is placedinto and transmitted together with encoded data in the predictionencoding process to be performed at step S123 hereinafter described.

At step S122, in the image conversion apparatus 322 (FIG. 31), thestorage section 351 of the coefficient acquisition section 341 (FIG. 32)updates the tap coefficients for the individual classes stored in thestorage section 351 into the tap coefficients for the individualintegration classes as the reduction filter information from thereduction apparatus 321 (stores the tap coefficients in the form thatthe tap coefficients for the individual integration classes as thereduction filter information are overwritten), and the processingadvances to step S123.

At step S123, a prediction encoding process of the original image isperformed, and the encoding process comes to an end.

FIG. 34 is a flow chart illustrating an example of the predictionencoding process at step S123 of FIG. 33.

In the prediction encoding process, processes similar to those at stepsS31 to S46 of FIG. 18 are performed at steps S131 to S146, respectively.

It is to be noted that, at step S142, the classification adaptive filter311 performs a classification adaptive process as a process of an ILFfor a decoding in-progress image from the arithmetic operation section110 similarly as at step S42 of FIG. 18. However, in the classificationadaptive process, tap coefficients for the individual integrationclasses and a corresponding relationship LUT as the reduction filterinformation to be outputted from the reduction apparatus 321 to theimage conversion apparatus 322 are used at step S120 of FIG. 33.

Further, at step S144, the reversible encoding section 106 encodes thequantization coefficients, encoding information and reduction filterinformation similarly as at step S44 of FIG. 18. However, the reductionfilter information includes the tap coefficients for the individualintegration classes and the corresponding relationship LUT.

Accordingly, the encoded data obtained by the reversible encodingsection 106 include the quantization coefficients, encoding information,and tap coefficients for the individual integration classes andcorresponding relationship LUT as the reduction filter information.Then, such encoded data are read out from the accumulation buffer 107and transmitted suitably as described in connection with step S45 ofFIG. 18 at step S145.

FIG. 35 is a flow chart illustrating an example of the classificationadaptive process performed at step S142 of FIG. 34.

In the image conversion apparatus 322 (FIG. 31) of the classificationadaptive filter 311, at step S151, the tap selection section 21 selectsone of pixels that have not been selected as a noticed pixel as yet as anoticed pixel from within a decoding in-progress image supplied from thearithmetic operation section 110 similarly as at step S51 of FIG. 19,and the processing advances to step S152.

At step S152, the tap selection sections 21 and 22 select pixels to bemade a prediction tap and a class tap regarding the noticed pixel fromwithin the decoding in-progress image supplied from the arithmeticoperation section 110 similarly as at step S52 of FIG. 19 and supply theselected pixels to the prediction arithmetic operation section 25 andthe classification section 23, respectively.

Thereafter, the processing advances from step S152 to step S153, atwhich the classification section 23 performs classification of thenoticed pixel using the class tap regarding the noticed pixel and theencoding information regarding the noticed pixel similarly as at stepS53 of FIG. 19.

Then, the classification section 23 supplies an initial class of thenoticed pixel obtained by classification of the noticed pixel to thecoefficient acquisition section 341, and the processing advances fromstep S153 to step S154.

At step S154, the integration class conversion section 352 of thecoefficient acquisition section 341 (FIG. 32) converts the initial classof the noticed pixel supplied from the classification section 23 into anintegration class of the noticed pixel in accordance with thecorresponding relationship LUT as the reduction filter informationsupplied from the reduction apparatus 321 at step S120 of FIG. 33. Then,the integration class conversion section 352 supplies the integrationclass of the noticed pixel to the acquisition section 353 (FIG. 32), andthe processing advances from step S154 to step S155.

At step S155, the acquisition section 353 acquires the tap coefficientsof the integration class of the noticed pixel from the integration classconversion section 352 from among the tap coefficients for theindividual integration classes stored at step S122 of FIG. 33 into thestorage section 351 and supplies the acquired tap coefficients to theprediction arithmetic operation section 25, and the processing advancesto step S156.

At steps S156 to S158, processes similar to those at steps S55 to S57 ofFIG. 19 are performed, respectively.

As described above, the encoding apparatus 11 of FIG. 27 transmits notthe tap coefficients for the individual initial classes obtained by tapcoefficient learning but the tap coefficients for the individualintegration classes where the initial classes are integrated into thenumber of classes smaller than the number of the initial classes and thecorresponding relationship LUT as the corresponding relationshipinformation between the initial classes and the integration classes asreduction filter information.

Further, integration of the tap coefficients for the individual initialclasses is performed in response to the RD costs as tap coefficientevaluation values each representative of appropriateness in use of thetap coefficients for each integration class in the classificationadaptive process.

Accordingly, by integrating initial classes into a smaller number ofintegration classes, the compression efficiency and the S/N of thedecoded image can be improved in comparison with those in an alternativecase in which tap coefficients for individual initial classes aretransmitted.

<Second Configuration Example of Decoding Apparatus 12>

FIG. 36 is a clock diagram depicting a second configuration example ofthe decoding apparatus 12 of FIG. 1.

It is to be noted that, in FIG. 36, elements corresponding to those ofFIG. 20 are denoted by like reference numerals and description of themis hereinafter omitted suitably.

Referring to FIG. 36, the decoding apparatus 12 includes the componentsfrom the accumulation buffer 201 to the arithmetic operation section205, the sorting buffer 207, the D/A conversion section 208, thecomponents from the frame memory 210 to the selection section 214, and aclassification adaptive filter 411.

Accordingly, the decoding apparatus 12 of FIG. 36 is common to that ofFIG. 20 in that it includes the components from the accumulation buffer201 to the arithmetic operation section 205, the sorting buffer 207, theD/A conversion section 208 and the components from the frame memory 210to the selection section 214.

However, the decoding apparatus 12 of FIG. 36 is different from that ofFIG. 20 in that it includes the classification adaptive filter 411 inplace of the classification adaptive filter 206.

The decoding apparatus 12 of FIG. 36 decodes encoded data transmittedfrom the encoding apparatus 11 of FIG. 27.

Therefore, the reduction filter information supplied from the reversibledecoding section 202 to the classification adaptive filter 411 includestap coefficients for individual integration classes and a correspondingrelationship LUT.

The classification adaptive filter 411 is a filter that functions as anILF by performing a classification adaptive process and is common to theclassification adaptive filter 206 of FIG. 20 in that it performs an ILFprocess by a classification adaptive process.

However, the classification adaptive filter 411 is different from theclassification adaptive filter 206 in that it performs theclassification adaptive process using the tap coefficients for theindividual integration classes and the corresponding relationship LUT asthe reduction filter information.

<Configuration Example of Classification Adaptive Filter 411>

FIG. 37 is a block diagram depicting a configuration example of theclassification adaptive filter 411 of FIG. 36.

Referring to FIG. 37, the classification adaptive filter 411 includes animage conversion apparatus 431.

To the image conversion apparatus 431, a decoding in-progress image issupplied from the arithmetic operation section 205 (FIG. 36), and tapcoefficients for individual integration classes and a correspondingrelationship LUT as reduction filter information and encodinginformation are supplied from the reversible decoding section 202.

The image conversion apparatus 431 performs image conversion by aclassification adaptive process using tap coefficients for individualintegration classes (tap coefficients obtained using reduction filterinformation) and a corresponding relationship LUT as reduction filterinformation using a decoding in-progress image as a first image toconvert the decoding in-progress image as the first information into anafter-filter image as a second image equivalent to an original image(generates an after-filter image), and supplies the after-filter imageto the sorting buffer 207 and the frame memory 210 (FIG. 36) similarlyto the image conversion apparatus 322 of FIG. 28.

It is to be noted that the image conversion apparatus 431 performs, inthe classification adaptive process, the classification using theencoding information as occasion demands similarly to the imageconversion apparatus 322 of FIG. 28.

<Configuration Example of Image Conversion Apparatus 431>

FIG. 38 is a block diagram depicting a configuration example of theimage conversion apparatus 431 of FIG. 37.

It is to be noted that, in FIG. 38, elements corresponding to those ofthe image conversion apparatus 231 of FIG. 22 are denoted by likereference numerals and description of them is hereinafter omittedsuitably.

Referring to FIG. 38, the image conversion apparatus 431 includes thecomponents from the tap selection section 241 to the classificationsection 243, the prediction arithmetic operation section 245 and acoefficient acquisition section 441.

Accordingly, the image conversion apparatus 431 of FIG. 38 is common tothe image conversion apparatus 231 of FIG. 22 in that it includes thecomponents from the tap selection section 241 to the classificationsection 243 and the prediction arithmetic operation section 245.

However, the image conversion apparatus 431 of FIG. 38 is different fromthe image conversion apparatus 231 of FIG. 22 in that it includes thecoefficient acquisition section 441 in place of the coefficientacquisition section 244.

To the coefficient acquisition section 441, tap coefficients forindividual integration classes as reduction filter information and acorresponding relationship LUT are supplied from the reversible decodingsection 202 (FIG. 36).

The coefficient acquisition section 441 acquires, setting tapcoefficients for individual integration classes as reduction filterinformation from the reversible decoding section 202 as tap coefficientsfor individual classes to be used in a classification adaptive process,tap coefficients of an integration class of a noticed pixel from the tapcoefficients for the individual integration classes, and supplies theacquired tap coefficient to the prediction arithmetic operation section245.

In particular, the coefficient acquisition section 441 converts a class(initial class) of the noticed pixel from the classification section 243into an integration class in accordance with the correspondingrelationship LUT as reduction filter information from the reversibledecoding section 202. Further, the coefficient acquisition section 441acquires tap coefficients of the integration class of the noticed pixelfrom the tap coefficients for the individual integration classes as thereduction filter information, and supplies the acquired tap coefficientto the prediction arithmetic operation section 245.

FIG. 39 is a block diagram depicting a configuration example of thecoefficient acquisition section 441 of FIG. 38.

Referring to FIG. 39, the coefficient acquisition section 441 includes astorage section 451, an integration class conversion section 452 and anacquisition section 453. The storage section 45 to the acquisitionsection 453 are configured similarly to the components from the storagesection 351 to the acquisition section 353 of FIG. 32, respectively.

In particular, to the storage section 451, tap coefficients for eachintegration class as reduction filter information are supplied from thereversible decoding section 202 (FIG. 36).

The storage section 451 stores the tap coefficients for individualintegration classes as reduction filter information from the reversibledecoding section 202.

To the integration class conversion section 452, an initial class as aclassification result of classification of a noticed pixel is suppliedfrom the classification section 243. Further, to the integration classconversion section 452, a corresponding relationship LUT as reductionfilter information is supplied from the reversible decoding section 202.

The integration class conversion section 452 converts the initial classof the noticed pixel into an integration class of the noticed pixel inaccordance with the corresponding relationship LUT and supplies theintegration class to the acquisition section 453.

The acquisition section 453 acquires tap coefficients of the integrationclass of the noticed pixel from the integration class conversion section452 from among the tap coefficients for the individual integrationclasses stored in the storage section 451, and then supplies the tapcoefficient to the prediction arithmetic operation section 245.

<Decoding Process>

FIG. 40 is a flow chart illustrating an example of a decoding process ofthe decoding apparatus 12 of FIG. 36.

In the decoding process, at step S171, the accumulation buffer 201temporarily accumulates encoded data transmitted thereto from theencoding apparatus 11 similarly as at step S71 of FIG. 24 and suitablysupplies the encoded data to the reversible decoding section 202, andthe processing advances to step S172.

At step S172, the reversible decoding section 202 receives and decodesthe encoded data supplied from the accumulation buffer 201 similarly asat step S72 of FIG. 24, and supplies a quantization coefficient obtainedby the decoding to the dequantization section 203.

Further, in the case where encoding information and reduction filterinformation are obtained by the decoding of the encoded data, thereversible decoding section 202 supplies necessary encoding informationto the intra-prediction section 212, motion prediction compensationsection 213 and other necessary blocks.

Further, the reversible decoding section 202 supplies the encodinginformation and the reduction filter information to the classificationadaptive filter 411.

Thereafter, the processing advances from step S172 to step S173, atwhich the classification adaptive filter 411 decides whether reductionfilter information is supplied from the reversible decoding section 202.

In the case where it is decided at step S173 that reduction filterinformation is not supplied, the processing advances to step S175skipping step S174.

On the other hand, in the case where it is decided at step S173 thatreduction filter information is supplied, the processing advances tostep S174, at which the coefficient acquisition section 441 configuringthe image conversion apparatus 431 (FIG. 38) of the classificationadaptive filter 411 acquires the tap coefficients for individualintegration classes as reduction filter information and a correspondingrelationship LUT.

Further, in the coefficient acquisition section 441 (FIG. 39), thestorage section 451 updates the tap coefficients for the individualclasses stored in the storage section 451 into the tap coefficients forthe individual integration classes as the reduction filter information(stores in such a form that the tap coefficients for the individualintegration classes as the reduction filter information areoverwritten).

Then, the processing advances from step S174 to step S175, at which aprediction decoding process is performed, and the decoding process isended.

FIG. 41 is a flow chart illustrating an example of the predictiondecoding process at step S175 of FIG. 40.

In the prediction decoding process, at steps S181 to S189, processessimilar to those at steps S81 to S89 of FIG. 25 are performed,respectively.

It is to be noted that, although, at step S186, the classificationadaptive filter 411 performs a classification adaptive process as aprocess of an ILF for a decoding in-progress image from the arithmeticoperation section 205 similarly as at step S86 of FIG. 25, in theclassification adaptive process, tap coefficients for individualintegration classes and a corresponding relationship LUT as reductionfilter information acquired by the coefficient acquisition section 441at step S174 of FIG. 40 are used.

FIG. 42 is a flow chart illustrating an example of the classificationadaptive process performed at step S186 of FIG. 41.

In the image conversion apparatus 431 (FIG. 38) of the classificationadaptive filter 411, at step S191, the tap selection section 241 selectsone of pixels that have not been made a noticed pixel as yet from withinthe decoding in-progress image supplied from the arithmetic operationsection 205 similarly as at step S91 of FIG. 26, and the processingadvances to step S192.

At step S192, the tap selection sections 241 and 242 select pixels to bemade a prediction tap and a class tap regarding a noticed pixel from thedecoding in-progress image supplied from the arithmetic operationsection 205 similarly as at step S92 of FIG. 26 and supplies the pixelsto the prediction arithmetic operation section 245 and theclassification section 243, respectively.

Thereafter, the processing advances from step S192 to step S193, atwhich the classification section 243 performs classification of thenoticed pixel using the class tap regarding the noticed pixel and theencoding information regarding the noticed pixel similarly as at stepS93 of FIG. 26.

Then, the classification section 243 supplies an initial class of thenoticed pixel obtained by classification of the noticed pixel to thecoefficient acquisition section 441, and the processing advances fromstep S193 to step S194.

At step S194, the integration class conversion section 452 of thecoefficient acquisition section 441 (FIG. 39) converts the initial classof the noticed pixel supplied from the classification section 243 intoan integration class of the noticed pixel in accordance with thecorresponding relationship LUT as the reduction filter informationacquired by the coefficient acquisition section 441 at step S174 of FIG.40. Then, the integration class conversion section 452 supplies theintegration class of the noticed pixel to the acquisition section 453(FIG. 39), and the processing advances from step S194 to step S195.

At step S195, the acquisition section 453 acquires tap coefficients ofthe integrated class of the noticed pixel from the integration classconversion section 452 from among the tap coefficients for theindividual integration classes stored at step S174 of FIG. 40 into thestorage section 451, and supplies the tap coefficients to the predictionarithmetic operation section 245. Then, the processing advances to stepS196.

At steps S196 to S198, processes similar to those at steps S95 to S97 ofFIG. 26 are performed, respectively.

Since the encoding apparatus 11 of FIG. 27 and the decoding apparatus 12of FIG. 36 perform an ILF process by a classification adaptive processin such a manner as described above, an after-filter image closer to theoriginal image than a process result of an ILF can be obtained. As aresult, the S/N of the decoded image can be improved. Further, since anafter-filter image close to the original image can be obtained, theresidual becomes small, and therefore, the compression efficiency can beimproved. Further, in the encoding apparatus 11, tap coefficients ofinitial classes are integrated such that the tap coefficient evaluationvalue is improved to generate reduction filter information in which tapcoefficients for each class are reduced, and not tap coefficients forthe individual initial classes but reduction filter information istransmitted to the decoding apparatus 12. Therefore, the compressionefficiency can be improved further.

<Third Configuration Example of Encoding Apparatus 11>

FIG. 43 is a block diagram depicting a third configuration example ofthe encoding apparatus 11 of FIG. 1.

It is to be noted that, in FIG. 43, elements corresponding to those ofFIG. 9 are denoted by like reference numerals and description of them ishereinafter omitted suitably.

Referring to FIG. 43, the encoding apparatus 11 includes the componentsfrom the A/D conversion section 101 to the arithmetic operation section110 and from the frame memory 112 to the rate controlling section 117,and a classification adaptive filter 511.

Accordingly, the encoding apparatus 11 of FIG. 43 is common to that ofFIG. 9 in that it includes the components from the A/D conversionsection 101 to the arithmetic operation section 110 and from the framememory 112 to the rate controlling section 117.

However, the encoding apparatus 11 of FIG. 43 is different from that ofFIG. 9 in that it includes the classification adaptive filter 511 inplace of the classification adaptive filter 111.

The classification adaptive filter 511 is a filter that functions as anILF by executing a classification adaptive process, and is common to theclassification adaptive filter 111 in that is performs an ILF process bythe classification adaptive process.

However, the classification adaptive filter 511 is different from theclassification adaptive filter 111 in that, in the classificationadaptive filter 511, a seed coefficient for each class ((β_(m,n)) of theexpression (9)) is generated as reduction filter information in thereduction process for generating reduction filter information thatreduces tap coefficients for each class.

Further, the classification adaptive filter 511 is different from theclassification adaptive filter 111 in that it performs a classificationadaptive process using tap coefficients for each class obtained using aseed coefficient for each class as reduction filter information.

In particular, the classification adaptive filter 511 is different fromthe classification adaptive filter 111 in that tap coefficients for eachclass is generated from a seed coefficient for each class as reductionfilter information.

Further, the classification adaptive filter 511 is different from theclassification adaptive filter 111 in that, in the classificationadaptive filter 511, not only reduction filter information but alsoparameter information are supplied to the reversible encoding section106 and transmitted.

In particular, as described hereinabove with reference to FIGS. 5 to 8,although, for example, a parameter z is necessitated to generate tapcoefficients from a seed coefficient, the classification adaptive filter511 supplies parameter information relating to the parameter z to beused to generate tap coefficients from a seed coefficient to thereversible encoding section 106.

Consequently, the parameter information is placed into and transmittedtogether with, for example, encoded data similarly to reduction filterinformation.

It is to be noted that, as described hereinabove with reference to FIGS.5 to 8, as the parameter that is used when tap coefficients aregenerated from a seed coefficient, in addition to one (kind of)parameter z, a plurality of parameters such as two parameters z_(x) andz_(y) or three parameters z_(x), z_(y) and z_(t). However, in thefollowing description, in order to simplify description, it is assumedthat one parameter z is adopted as the parameter to be used when tapcoefficients are generated from a seed coefficient.

<Configuration Example of Classification Adaptive Filter 511>

FIG. 44 is a block diagram depicting a configuration example of theclassification adaptive filter 511 of FIG. 43.

Referring to FIG. 44, the classification adaptive filter 511 includes alearning apparatus 531 and an image conversion apparatus 322.

To the learning apparatus 531, an original image is supplied from thesorting buffer 102 (FIG. 43) and a decoding in-progress image issupplied from the arithmetic operation section 110 (FIG. 43). Further,encoding information is supplied to the learning apparatus 531.

The learning apparatus 531 uses the decoding in-progress image asstudent data and uses the original image as teacher data to performlearning for determining a seed coefficient for each class (hereinafterreferred to also as seed coefficient learning).

Further, the learning apparatus 531 supplies seed coefficients forindividual classes obtained by seed coefficient learning as reductionfilter information that reduces tap coefficients for the individualclasses obtained by tap coefficient learning to the image conversionapparatus 532 and the reversible encoding section 106 (FIG. 43).

Here, according to the seed coefficient, since tap coefficients can bedetermined for the parameter z of various values can be determined inaccordance with the expression (9), it can be considered that the seedcoefficient is information that reduces the tap coefficients for theparameter z of the plurality of values (reduction filter information).

Accordingly, the learning apparatus 531 functions as a learningapparatus that determines seed coefficients for individual classes byseed coefficient learning and functions also as a reduction section thatreduces reduction filter information that reduces tap coefficients forindividual classes.

It is to be noted that the learning apparatus 531 performs, in the seedcoefficient learning, classification using the encoding information asoccasion demands.

To the image conversion apparatus 532, a decoding in-progress image issupplied from the arithmetic operation section 110 (FIG. 43) andreduction filter information is supplied from the learning apparatus531. Further, encoding information is supplied to the image conversionapparatus 532.

The image conversion apparatus 532 determines tap coefficients forindividual classes using seed coefficients for individual classes asreduction filter information of the learning apparatus 531. Further, theimage conversion apparatus 532 performs, setting the decodingin-progress image as a first image, image conversion by a classificationadaptive process in which the tap coefficients for the individualclasses are used to convert the decoding in-progress image as the firstimage into an after-filter image as a second image equivalent to theoriginal image (generates an after-filter image), and supplies theafter-filter image to the frame memory 112 (FIG. 43).

Further, when to determine tap coefficients for the individual classes,the image conversion apparatus 532 uses the parameter z in addition tothe seed coefficients for the individual classes as the reduction filterinformation, and supplies parameter information relating to theparameter z to the reversible encoding section 106 (FIG. 43).

It is to be noted that the image conversion apparatus 532 performs, inthe classification adaptive process, classification using the encodinginformation as occasion demands.

<Configuration Example of Learning Apparatus 531>

FIG. 45 is a block diagram depicting a configuration example of thelearning apparatus 531 of FIG. 44.

Referring to FIG. 45, the learning apparatus 531 includes a parametergeneration section 541, an order setting section 542, a learning section543 and a selection section 544.

To the parameter generation section 541, encoding information issupplied. The parameter generation section 541 generates a parameter zof a value, for example, according to encoding information of a noticedpixel (including encoding information of a block including the noticedpixel, encoding information of a frame (picture) including the noticedpixel and so forth) from within the encoding information suppliedthereto, and supplies the parameter z to the learning section 543.

Here, as the encoding information of the noticed pixel to be used forgeneration of a parameter z, encoding information suitable forimplementing an ILF upon encoding/decoding by a classification adaptiveprocess such as, for example, a code amount target value (bitrate) whenthe noticed pixel is to be encoded, a quantization parameter QP or thelike can be adopted.

In the case where a code amount target value or a quantization parameterQP is adopted as the encoding information to be used for generation of aparameter z, a parameter z of a value according to the code amounttarget value or the quantization parameter QP and eventually to thedesign (activity) of the original image is generated.

Here, the parameter generation section 541 not only can generate oneparameter z in response to a single piece of encoding information fromamong a plurality of pieces of encoding information such as a codeamount target value, a quantization parameter QP or the like but alsocan generate a plurality of parameters in response to individual ones ofa plurality of pieces of encoding information such as a code amounttarget value, a quantization parameter QP and so forth. However, asdescribed hereinabove with reference to FIG. 43, in the presentembodiment, in order to simplify description, a single parameter z isadopted as the parameter to be used when tap coefficients are generatedfrom a seed coefficient.

The order setting section 542 sets an order M of a seed coefficient tobe determined by seed coefficient learning, namely, a number M of seedcoefficients β_(m,n) to be used when tap coefficients w_(n) aredetermined in accordance with the expression (9) and supplies the numberM to the learning section 543.

To the learning section 543, a parameter z is supplied from theparameter generation section 541 and an order M is supplied from theorder setting section 542, and besides an original image is suppliedfrom the sorting buffer 102 (FIG. 43) and a decoding in-progress imageis supplied from the arithmetic operation section 110 (FIG. 43).Furthermore, to the learning section 543, encoding information issupplied.

The learning section 531 uses the decoding in-progress image as studentdata and uses the original image as teacher data, and uses the parameterz generated by the parameter generation section 541 to perform seedcoefficient learning for determining a seed coefficient of the order Mfrom the order setting section 542.

The learning section 531 determines seed coefficients of a plurality oforders M, for example, for each class by seed coefficient learning andsupplies the seed coefficients to the selection section 544.

It is to be noted that the learning section 531 performs, in the seedcoefficient learning, classification using the encoding information asoccasion demands.

The selection section 544 selects, for each class, a seed coefficient ofa predetermined order from among the seed coefficients of the pluralityof orders M from the learning section 531 and supplies the selected seedcoefficient as reduction filter information to the image conversionapparatus 532 (FIG. 44) and the reversible encoding section 106 (FIG.43).

FIG. 46 is a block diagram depicting a configuration example of thelearning section 543 of FIG. 45.

It is to be noted that, in FIG. 46, elements corresponding to those ofthe learning section 63 of FIG. 8 are denoted by like reference numeralsand description of them is hereinafter omitted suitably.

Further, while, in FIG. 46, the learning section 543 is configuredutilizing the learning section 63 of FIG. 8, the learning section 543can otherwise be configured utilizing the learning section 63 of FIG. 7.

Referring to FIG. 46, the learning section 543 includes the componentsfrom the tap selection section 41 to the classification section 43, thecoefficient calculation section 45 and the components from the additionsection 81 to the coefficient calculation section 83.

Accordingly, the learning section 543 is configured similarly to thelearning section 63 of FIG. 8.

However, in the learning section 543, the classification section 43performs classification using a class tap or encoding information asoccasion demands.

Further, in the learning section 543, a decoding in-progress image isused as student data and an original image corresponding to the decodingin-progress image is used as teacher data to perform seed coefficientlearning, and seed coefficients for individual classes obtained by theseed coefficient learning are supplied from the coefficient calculationsection 83 to the selection section 544 (FIG. 45).

Further, in the learning section 543, the addition section 82 performs,for each class, addition of (a variable t_(m) corresponding to) aparameter z supplied from the parameter generation section 541 (FIG. 45)and an optimum tap coefficient w_(n) supplied from the coefficientcalculation section 45 similarly as in the case described hereinabovewith reference to FIG. 8. However, the addition is performed for theplurality of orders M supplied from the order setting section 542. Inparticular, calculation of the expression (26) is performed for theplurality of orders M from the order setting section 542.

As a result, in the addition section 82, a normal equation of theexpression (28) is determined for the plurality of orders M and issupplied to the coefficient calculation section 83. Accordingly, in thecoefficient calculation section 83, a seed coefficient is determined inregard to the plurality of orders M for each class.

It is to be noted that seed coefficient learning by the learningapparatus 531 (FIG. 44) not only can be performed in parallel toencoding of an original image but also can be performed in advanceirrespective of encoding of an original image similarly to the tapcoefficient learning of the learning apparatus 131 of FIG. 10.

<Relationship Between Parameter z and Tap Coefficient>

FIG. 47 is a view illustrating a relationship between a parameter z andtap coefficients.

In FIG. 47, on a two-dimensional image defined by the axis of abscissaindicating the parameter z and the axis of ordinate indicating the tapcoefficient, coefficient points representative of tap coefficients withrespect to the parameter z, namely, tap coefficients obtained by tapcoefficient learning using an original image and a decoding in-progressimage from which a parameter z is obtained are plotted.

Although, in FIG. 47, a plurality of coefficient points, namely,coefficient points of tap coefficients with respect to a plurality ofvalues of the parameter z, are plotted, a curve representative of arelationship between the parameter z and the tap coefficientscorresponding to the plurality of points is hereinafter referred to asrelationship curve.

The relationship curve is represented by the expression (9) in which aseed coefficient is used, and accordingly, the seed coefficientprescribes (defines) the relationship curve.

For example, in the case where the parameter z has a value according toa design (activity) of an original image as described hereinabove withreference to FIG. 45, as the relationship curve defined by the seedcoefficient fits more with the distribution of coefficient points, animage close to the original image (image having a high S/N) is obtainedby the classification adaptive process in which tap coefficientsobtained from the seed coefficient and the parameter z are used isobtained in regard to an image of a design according to the parameter z.

FIG. 48 is a view illustrating an example of a relationship between thedistribution of coefficient points and the order of the seed coefficientthat defines a relationship curve that fits with the distribution of thecoefficient points.

In the case where the distribution of coefficient points is a simpledistribution close to a linear shape, the seed coefficient that definesa relationship curve that fits with the distribution of coefficientpoints may be a seed coefficient of a low order.

On the other hand, in the case where the distribution of coefficientpoints is complicated, the seed coefficient that defines a relationshipcurve that fits with a distribution of coefficient points is a seedcoefficient of a great (high) order.

FIG. 49 is a view illustrating another example of a relationship betweena distribution of coefficient points and an order of a seed coefficientthat defines a relationship curve that fits with the distribution ofcoefficient points.

As described above with reference to FIG. 48, in the case where thedistribution of coefficient points is complicated, the seed coefficientthat defines a relationship curve that fits with the distribution ofcoefficient points is a seed coefficient of a high order.

However, within a certain interval D (in the time direction) of anoriginal image, even in the case where the distribution of coefficientsis complicated, the deflection width (fluctuation) of (encodinginformation whose generation is based on) the parameter z is sometimessmaller within some interval D′ of the interval D.

In this case, although the order of a seed coefficient that defines arelationship curve that fits with the distribution of coefficient pointsof the tap coefficient in which the original image of the interval Dbecomes high, the order of the seed coefficient that defines therelationship curve that fits with the distribution of the coefficientpoints of the tap coefficient in which the original image of theinterval D′ can be made low.

Therefore, in the learning apparatus (FIG. 45), the selection section544 selects a seed coefficient of an order that defines a relationshipcurve that fits with the distribution of coefficient points from amongseed coefficients of a plurality of orders obtained by seed coefficientlearning by the learning section 531 in response to a seed coefficientevaluation value representative of appropriateness in use of tapcoefficients determined from a seed coefficient in (the predictionarithmetic operation of the expression (1) of) the classificationadaptive process, and supplies the selected seed coefficients asreduction filter information to the image conversion apparatus 532 (FIG.44) and the reversible encoding section 106 (FIG. 43).

Here, as the seed coefficient evaluation value, for example, a valuecorresponding to the RD cost when an original image to be used in seedcoefficient learning is encoded can be adopted. For example, it ispossible to obtain a seed coefficient of a low order that defines arelationship curve that fits with the distribution of coefficient pointsas reduction filter information by adopting the RD cost as the seedcoefficient evaluation value and selecting a seed coefficient of anorder whose RD cost is best from among seed coefficients of a pluralityof orders obtained by seed coefficient learning as reduction filterinformation. As a result, the compression efficiency and the S/N of thedecoded image can be improved.

Further, as the seed coefficient evaluation value, for example, a valuecorresponding to a design (activity) of an original image used in seedcoefficient learning can be adopted. For example, by adopting avariation width of a design (for example, a variation amount ofactivity) within a predetermined interval of an original image as theseed coefficient evaluation value and selecting a seed coefficient ofsuch an order that increases in proportion to the variation width of thedesign from among seed coefficients of a plurality of orders obtained bythe seed coefficient learning, it is possible to obtain a seedcoefficient of a lower order that defines a relationship curve that fitswith the distribution of coefficient points of the tap coefficientsappropriate for a classification adaptive process of a decodingin-progress image corresponding to an original image of various designsas reduction filter information. As a result, the compression efficiencyand the S/N of the decoded image can be improved.

Furthermore, as the seed coefficient evaluation value, for example, avalue corresponding to encoding information such as a code amount targetvalue (bitrate), a quantization parameter QP or the like when anoriginal image to be used in seed coefficient learning is to be encodedcan be adopted. For example, by adopting a variation width of a codeamount target value or a quantization parameter QP within apredetermined interval of an original image as the seed coefficientevaluation value and selecting a seed coefficient of such an order thatincreases in proportion to the variation width of the code amount targetvalue or the quantization parameter QP from among seed coefficients of aplurality of orders obtained by the seed coefficient learning, it ispossible to obtain a seed coefficient of a lower order that defines arelationship curve that fits with the distribution of coefficient pointsof the tap coefficients appropriate for a classification adaptiveprocess of a decoding in-progress image corresponding to an originalimage of various code amount target values or quantization parameters QPas reduction filter information. As a result, the compression efficiencyand the S/N of the decoded image can be improved.

Further, as the seed coefficient evaluation value, for example, a valuecorresponding to a parameter z generated for an original image used inseed coefficient learning can be adopted. For example, by adopting avariation width (deflection width) of the parameter z within an intervalof an original image to be used in seed coefficient learning andselecting a seed coefficient of such an order that increases inproportion to the variation width of the parameter z among seedcoefficients of a plurality of orders obtained by the seed coefficientlearning as reduction filter information, it is possible to obtain aseed coefficient of a lower order that defines a relationship curve thatfits with the distribution of coefficient points as reduction filterinformation. As a result, the compression efficiency and the S/N of thedecoded image can be improved.

From the point of view of improvement of the compression efficiency andimprovement of the S/N of the decoded image, it is desirable to adopt avalue corresponding to the RD cost as the seed coefficient evaluationvalue. However, in the case where the RD cost is adopted as the seedcoefficient evaluation value, a high calculation cost is required forcalculation of the seed coefficient evaluation value. By adopting, asthe seed coefficient evaluation value, a value corresponding, forexample, to the design of the original image described hereinabove, tothe encoding information or to the parameter z, the calculation costrequired for calculation of the seed coefficient evaluation value can bereduced.

Here, in the case where the design of the original image and eventuallythe code amount target value or the quantization parameter QP, forexample, upon encoding of the original image change temporally, in orderto obtain an after-filter image closer to the original image by aclassification adaptive process in regard to the original image afterthe change, it is desirable to perform tap coefficient learning usingthe original image after the change and perform a classificationadaptive process using tap coefficients obtained by the tap coefficientlearning.

However, even in the case where the design of the original image changestemporally, if, when the change is not a very great change, tapcoefficient learning using the original image after the change isperformed and tap coefficients obtained by the tap coefficient learningare transmitted from the encoding apparatus 11 to the decoding apparatus12, then the compression efficiency degrades.

Further, in the case where the design of the original image changes by agreat amount as a result of a scene change or the like, if tapcoefficients obtained by tap coefficient learning using the originalimage before the change continues to be used in the classificationadaptive process, then the S/N of the decoded image (after-filter image)degrades.

In contrast, with a seed coefficient obtained by seed coefficientlearning using a parameter z generated in response to a code amounttarget value or a quantization parameter QP, it is possible todetermine, using a parameter z generated in response to a code amounttarget value or a quantization parameter QP upon encoding of theoriginal image, tap coefficients with which a decoded image(after-filter image) having a reduced error from the original imagecorresponding to the code amount target value or the quantizationparameter QP can be obtained. Then, by performing a classificationadaptive process using such tap coefficients as described above,degradation of the S/N of the decoded image can be prevented.

Further, with the seed coefficient, roughly it is possible to obtain adecoded image (after-filter image) of a high S/N in regard to anoriginal image within a range of the code amount target value or thequantization parameter QP with which a parameter z used in seedcoefficient learning for determining the seed coefficient is generated.In particular, in the case where the code amount target value or thequantization parameter QP upon encoding of the original imagetemporarily varies a little, even if the seed coefficient at presentcontinues to be used in the classification adaptive process, a decodedimage of a high S/N can be obtained. Accordingly, even if the codeamount target value or the quantization parameter QP upon encoding ofthe original image temporarily varies a little, since there is nonecessity to newly perform seed coefficient learning and transmit a newseed coefficient from the encoding apparatus 11 to the decodingapparatus 12, the compression efficiency can be improved.

As described above, with the seed coefficient, even if the design of theoriginal image and eventually the code amount target value or thequantization parameter QP upon encoding of the original image temporallyvaries a little, since a decoded image of a high S/N can be obtained bya classification adaptive process in which tap coefficients obtainedfrom the seed coefficient are used, it can be considered that the seedcoefficient absorbs the design of the original image, namely, forexample, a change in time of the code amount target value, thequantization parameter QP or the like upon encoding of the originalimage.

<Configuration Example of Image Conversion Apparatus 532>

FIG. 50 is a block diagram depicting a configuration example of theimage conversion apparatus 532 of FIG. 44.

It is to be noted that, in FIG. 50, elements corresponding to those ofthe image conversion apparatus 133 of FIG. 15 are denoted by likereference numerals and description of them is omitted suitably.

Referring to FIG. 50, the image conversion apparatus 532 includes thecomponents from the tap selection section 21 to the classificationsection 23, the prediction arithmetic operation section 25, a parametergeneration section 561 and a coefficient acquisition section 562.

Accordingly, the image conversion apparatus 532 is configured similarlyto the image conversion apparatus 133 of FIG. 15 in that it includes thecomponents from the tap selection section 21 to the classificationsection 23 and the prediction arithmetic operation section 25.

However, the image conversion apparatus 532 is different from the imageconversion apparatus 133 in that it additionally includes the parametergeneration section 561. The image conversion apparatus 532 is furtherdifferent from the image conversion apparatus 133 in that it includesthe coefficient acquisition section 562 in place of the coefficientacquisition section 151.

To the parameter generation section 561, encoding information issupplied. The parameter generation section 561 generates a parameter zsimilar to that of the parameter generation section 541 of the learningapparatus 531 of FIG. 45 in response to a code amount target value of anoticed pixel, a quantization parameter QP or the like from within theencoding information supplied thereto, and supplies the parameter z tothe coefficient acquisition section 562.

Further, the parameter generation section 561 supplies, for example, theparameter z as parameter information to the reversible encoding section106 (FIG. 43).

Here, as the parameter information to be applied from the parametergeneration section 561 to the reversible encoding section 106, not onlythe parameter z itself but also encoding information such as, forexample, a code amount target value, a quantization parameter QP or thelike used to generate the parameter z or the like can be adopted. In thecase where the parameter generation section 561 generates the parameterz using the encoding information supplied to the reversible encodingsection 106, the parameter information may not be supplied (transmitted)to the reversible encoding section 106.

To the coefficient acquisition section 562, not only the parameter z issupplied from the parameter generation section 561, but also a seedcoefficient for each class as reduction filter information is suppliedfrom the learning apparatus 531.

The coefficient acquisition section 562 generates tap coefficients foreach class in accordance with the expression (9) using the seedcoefficient for each class as the reduction filter information from thelearning apparatus 531 and the parameter z from the parameter generationsection 561. Then, the coefficient acquisition section 562 acquires thetap coefficient of the class of the noticed pixel supplied from theclassification section 23 from among the tap coefficients for theindividual classes and supplies the acquired tap coefficient to theprediction arithmetic operation section 25.

As an alternative, the coefficient acquisition section 562 uses a seedcoefficient of a class of a noticed pixel supplied from theclassification section 23 from among seed coefficients for individualclasses as reduction filter information from the learning apparatus 531and a parameter z from the parameter generation section 561 to acquiretap coefficients of the class of the noticed pixel by generation of thesame, and supplies the tap coefficients to the prediction arithmeticoperation section 25.

FIG. 51 is a block diagram depicting a configuration example of thecoefficient acquisition section 562 of FIG. 50.

Referring to FIG. 51, the coefficient acquisition section 562 includes astorage section 571, a tap coefficient calculation section 572, anotherstorage section 573 and an acquisition section 574.

To the storage section 571, a seed coefficient for each class asreduction filter information is supplied from the learning apparatus 531(FIG. 44).

The storage section 571 stores the seed coefficient for each class asthe reduction filter information from the learning apparatus 531.

To the tap coefficient calculation section 572, a parameter z generatedfor a noticed pixel is supplied from the parameter generation section561. The tap coefficient calculation section 572 uses the seedcoefficients for the individual classes stored in the storage section571 and the parameter z from the parameter generation section 561 tocalculate tap coefficients for each class in accordance with theexpression (9), and supplies the calculated tap coefficients to thestorage section 573.

The storage section 573 stores the tap coefficients for the individualclasses from the tap coefficient calculation section 572.

To the acquisition section 574, a class of a noticed pixel is suppliedfrom the classification section 23.

The acquisition section 574 acquires tap coefficients of a class of anoticed pixel from the classification section 23 from the tapcoefficients for the individual classes stored in the storage section571 and supplies the acquired tap coefficient to the predictionarithmetic operation section 25.

<Encoding Process>

FIG. 52 is a flow chart illustrating an example of an encoding processof the encoding apparatus 11 of FIG. 43.

In the encoding apparatus 11, the learning apparatus 531 (FIG. 44) ofthe classification adaptive filter 511 temporarily stores a decodingin-progress image supplied thereto as student data and temporarilystores an original image corresponding to the decoding in-progress imageas teacher data.

At step S211, the learning apparatus 531 (FIG. 44) of the classificationadaptive filter 511 decides whether the timing at present is an updatetiming for a seed coefficient.

Here, the update timing for a seed coefficient can be determined inadvance like, for example, after every one or more frames, after everyone or more sequences, after every one or more slices, after one or morelines of a predetermined block such as a CTU or the like similarly tothe update timing of tap coefficients described hereinabove withreference to FIG. 17.

Further, as the update timing for a seed coefficient, not only aperiodical (fixed) timing such as a timing after every one or moreframes but also a dynamic timing such as a timing at which the S/N ofthe after-filter image becomes equal to or lower than a threshold valueor the like can be adopted similarly to the update timing of tapcoefficients described hereinabove with reference to FIG. 17.

In the case where it is decided at step S211 that the timing at presentis not an update timing for a seed coefficient, the processing advancesto step S220 skipping steps S212 to S219.

On the other hand, in the case where it is decided at step S211 that thetiming at present is an update timing for a seed coefficient, theprocessing advances to step S212, at which the learning apparatus 531determines tap coefficients.

In particular, the learning apparatus 531 (FIG. 45) performs tapcoefficient learning using, for example, a decoding in-progress imageand an original image stored within a period from the update timing inthe immediately preceding operation cycle to the update timing in thecurrent operation cycle as student data and teacher data, respectively,and the coefficient calculation section 45 (FIG. 46) determines tapcoefficients for each class in regard to each of a plurality of valuesof the parameter z generated from the decoding in-progress image fromthe encoding information of the decoding in-progress image and theoriginal image that became the student data and the teacher data.

Then, in the learning section 543 (FIG. 46) of the learning apparatus531, the coefficient calculation section 45 supplies the tapcoefficients for the individual classes in regard to the plurality ofvalues of the parameter z to the addition section 82, and the processingadvances from step S212 to step S213.

At step S213, the learning section 543 selects one of classes that havenot been selected as a noticed class as a noticed class from among allclasses whose tap coefficient has been determined at step S212, and theprocessing advances to step S214.

At step S214, the addition section 82 and the coefficient calculationsection 83 of the learning section 543 (FIG. 46) calculate a seedcoefficient of the noticed class using the tap coefficients of thenoticed class in regard to the plurality of values of the parameter zfrom among the tap coefficients for the individual classes regarding theplurality of values of the parameter z determined at step S212 and theplurality of values of the parameter z.

In calculation of a seed coefficient of the noticed class, in thelearning apparatus 531 (FIG. 45), a plurality of orders M are suppliedfrom the order setting section 542 to the learning section 543, and theaddition section 82 and the coefficient calculation section 83 of thelearning section 543 (FIG. 46) calculate a seed coefficient of thenoticed class for each of the plurality of orders M.

In the learning apparatus 531 (FIG. 45), the learning section 543supplies the seed coefficient of the noticed class in regard to each ofthe plurality of orders M to the selection section 544, and theprocessing advances from step S214 to step S215.

At step S215, the selection section 544 calculates, in regard to a seedcoefficient of a noticed class in regard to each of the plurality oforders M from the learning section 543, for example, an RD cost or thelike as a seed coefficient evaluation value representative ofappropriateness in use of tap coefficients determined from the seedcoefficient in a classification adaptive process. Then, the processingadvances to step S216.

At step S216, the selection section 544 selects the seed coefficient ofthe noticed class of the order M in which the seed coefficientevaluation value is best from among the seed coefficients of the noticedclass in each of the plurality of orders M as the seed coefficient ofthe optimum order M, and the processing advances to step S217.

At step S217, the learning section 543 (FIG. 45) decides whether allclasses whose tap coefficient has been determined at step S212 have beenmade a noticed class.

In the case where it is decided at step S217 that all classes have notbeen made a noticed class as yet, the processing returns to step S213,and thereafter, similar processes are repeated.

On the other hand, in the case where it is decided at step S217 that allclasses have been made a noticed class, the processing advances to stepS218, at which the selection section 544 (FIG. 45) outputs the seedcoefficient of the optimum order M selected in regard to each class atstep S216 as reduction filter information to the reversible encodingsection 106 (FIG. 43) and the image conversion apparatus 532 (FIG. 44).

Further, at step S218, the reversible encoding section 106 (FIG. 43)sets the reduction filter information from (the selection section 544of) the learning apparatus 531 as a transmission target, and theprocessing advances to step S219. The reduction filter information setas a transmission target is included into and transmitted together withencoded data in a prediction encoding process to be performed at stepS220 hereinafter described.

At step S219, in the image conversion apparatus 532 (FIG. 50), thestorage section 571 of the coefficient acquisition section 562 (FIG. 51)updates the storage substance thereof to the seed coefficients of theoptimum order M for each class as the reduction filter information from(the selection section 544 of) the learning apparatus 531 (stores theseed coefficients for the individual classes in an overlapping form),and the processing advances to step S220.

At step S220, a prediction encoding process of the original image isperformed, and the encoding process ends therewith.

FIG. 53 is a flow chart illustrating an example of the predictionencoding process at step S220 of FIG. 52.

In the prediction encoding process, processes similar to those at stepsS31 to S46 of FIG. 18 are performed at steps S231 to S246, respectively.

It is to be noted that, while, at step S242, the classification adaptivefilter 511 performs a classification adaptive process as a process of anILF for a decoding in-progress image from the arithmetic operationsection 110 similarly as at step S42 of FIG. 18, in the classificationadaptive process, tap coefficients for the individual classes obtainedfrom the seed coefficients for the individual classes as reductionfilter information stored in the storage section 571 of the coefficientacquisition section 562 (FIG. 51) at step S219 of FIG. 52 are used.

Further, in the classification adaptive process at step S242, parameterinformation of the parameter z generated from encoding information of anoticed pixel is supplied from the classification adaptive filter 511 tothe reversible encoding section 106 as hereinafter described.

Further, although, at step S244, the reversible encoding section 106encodes quantization coefficients, encoding information and reductionfilter information similarly as at step S44 of FIG. 18, the reductionfilter information includes seed coefficients for individual classes.

Further at step S244, the reversible encoding section 106 encodesparameter information supplied from the classification adaptive filter511.

Accordingly, the encoded data obtained by the reversible encodingsection 106 include quantization coefficients, encoding information,seed coefficients for individual classes as reduction filter informationand parameter information. Thus, such encoded data are suitably readout, at step S245, from the accumulation buffer 107 and transmitted asdescribed hereinabove in connection with step S45 of FIG. 18.

FIG. 54 is a flow chart illustrating an example of a classificationadaptive process performed at step S242 of FIG. 53.

In the image conversion apparatus 532 (FIG. 50) of the classificationadaptive filter 511, at step S251, the tap selection section 21 selectsone of pixels that have not been selected as a noticed pixel as anoticed pixel from within the decoding in-progress image supplied fromthe arithmetic operation section 110 similarly as at step S51 of FIG.19, and the processing advances to step S252.

At step S252, the parameter generation section 561 (FIG. 50) generates aparameter z from the encoding information of the noticed pixel andsupplies the parameter z to the coefficient acquisition section 562.Further, the parameter generation section 561 supplies, as parameterinformation relating to the parameter z, for example, the parameter zitself to the reversible encoding section 106. The parameter informationsupplied to the reversible encoding section 106 is encoded at step S244and transmitted at step S245 as described hereinabove with reference toFIG. 53.

After step S252, the processing advances to step S253, at which the tapcoefficient calculation section 572 of the coefficient acquisitionsection 562 (FIG. 51) calculates tap coefficients for each class usingthe seed coefficients for the individual classes stored in the storagesection 571 at step S219 of FIG. 52 and the parameter z supplied fromthe parameter generation section 561 at the immediately preceding stepS252. Further, the tap coefficient calculation section 572 stores thetap coefficients for the individual classes into the storage section573, and the processing advances from step S253 to step S254.

Here, the generation of a parameter z at step S252 and the calculationof tap coefficients at step S253 can be performed, for example, for eachframe.

In particular, at step S252, encoding information of the frame of thenoticed image is used to generate a parameter z common to the frame,namely, a parameter z in a unit of a frame, and at step S253, theparameter z in a unit of a frame can be used to calculate tapcoefficients for each class in a unit of a frame.

Here, the generation of a parameter z and the calculation of tapcoefficients using the parameter z can be performed in an arbitrary unitother than a unit of a frame such as, for example, a unit of a pixel, aunit of a block, a unit of a plurality of frames or the like.

However, if the generation of a parameter z and the calculation of tapcoefficients are performed in an excessively fine unit, then thefrequency in which parameter information of the parameter z istransmitted becomes high and the compression efficiency degrades. On theother hand, if the generation of a parameter z and the calculation oftap coefficients are performed in an excessively rough (great) unit,then there is the possibility that the S/N of the after-filter imageobtained by the classification adaptive process that uses the tapcoefficient may degrade.

Accordingly, the unit in which generation of a parameter z andcalculation of tap coefficients are performed is preferably determinedtaking the compression efficiency and the S/N into consideration.

Further, supply of parameter information to the reversible encodingsection 106, namely, transmission of parameter information, can beperformed not only every time a parameter z is generated but also onlywhen a parameter z different from that in the preceding operation cycleis generated (such transmission is not performed in the case where aparameter z same as that in the preceding operation cycle is generated).

By performing transmission of parameter information only in the casewhere a parameter z different from that in the preceding operation cycleis generated, degradation of the compression efficiency in the casewhere generation of a parameter z is performed in a fine unit can besuppressed.

Similarly, also calculation of tap coefficient using a parameter z canbe performed only in the case where a parameter z different from that inthe preceding operation cycle is generated. In this case, the arithmeticoperation cost required for calculation of tap coefficients can bereduced.

At step S254, the tap selection sections 21 and 22 select pixels to bemade a prediction tap and a class tap regarding a noticed image fromwithin the decoding in-progress image supplied from the arithmeticoperation section 110 similarly as at step S52 of FIG. 19 and supply thepixels to the prediction arithmetic operation section 25 and theclassification section 23, respectively.

Thereafter, the processing advances from step S254 to step S255, atwhich the classification section 23 performs classification of thenoticed pixel using the class tap regarding the noticed image and theencoding information regarding the noticed pixel similarly as at stepS53 of FIG. 19.

Then, the classification section 23 supplies an initial class of thenoticed pixel obtained by the classification of the noticed pixel to thecoefficient acquisition section 562, and the processing advances fromstep S255 to step S256.

At step S256, the acquisition section 574 of the coefficient acquisitionsection 562 (FIG. 51) acquires tap coefficients of the class of thenoticed pixel from among the tap coefficients for the individual classesstored in the storage section 573 at step S253 performed immediatelypreviously and supplies the acquired tap coefficients to the predictionarithmetic operation section 25. Then, the processing advances to stepS257.

At steps S257 to S259, processes similar to those at steps S55 to S57 ofFIG. 19, respectively, are performed.

As described above, in the encoding apparatus 11 of FIG. 43, not tapcoefficients for each class but a seed coefficient for each classobtained by seed coefficient learning is transmitted as the reductionfilter information.

Further, the order of a seed coefficient for each class is selected suchthat the seed coefficient evaluation value such as the RD cost becomesbest and has a lower value within a range within which a decoded image(after-filter image) of a high S/N is obtained.

Accordingly, the compression efficiency and the S/N of the decoded imagecan be improved.

<Third Configuration Example of Decoding Apparatus 12>

FIG. 55 is a block diagram depicting a third configuration example ofthe decoding apparatus 12 of FIG. 1.

It is to be noted that, in FIG. 55, elements corresponding to those ofFIG. 20 are denoted by like reference numerals and description of themis hereinafter omitted suitably.

Referring to FIG. 55, the decoding apparatus 12 includes the componentsfrom the accumulation buffer 201 to the arithmetic operation section205, the sorting buffer 207, the D/A conversion section 208, thecomponents from the frame memory 210 to the selection section 214, and aclassification adaptive filter 611.

Accordingly, the decoding apparatus 12 of FIG. 55 is common to that ofFIG. 20 in that it includes the components from the accumulation buffer201 to the arithmetic operation section 205, sorting buffer 207, D/Aconversion section 208 and components from the frame memory 210 to theselection section 214.

However, the decoding apparatus 12 of FIG. 55 is different from that ofFIG. 20 in that it includes the classification adaptive filter 611 inplace of the classification adaptive filter 206.

The decoding apparatus 12 FIG. 55 decodes encoded data transmitted fromthe encoding apparatus 11 of FIG. 43.

Therefore, reduction filter information supplied from the reversibledecoding section 202 to the classification adaptive filter 611 includesa seed coefficient for each class.

Further, encoded data transmitted from the encoding apparatus 11 of FIG.43 include parameter information. In the case where parameterinformation is obtained by decoding of encoded data by the reversibledecoding section 202, the parameter information is supplied from thereversible decoding section 202 to the classification adaptive filter611.

The classification adaptive filter 611 is a filter that functions as anILF by performing a classification adaptive process and is common to theclassification adaptive filter 206 of FIG. 20 in that it performs an ILFprocess by a classification adaptive process.

However, the classification adaptive filter 611 is different from theclassification adaptive filter 206 in that it calculates tapcoefficients for each class from a parameter z obtained from theparameter information and a seed coefficient for each class as reductionfilter information and performs a classification adaptive process usingthe tap coefficients for the individual classes.

<Configuration Example of Classification Adaptive Filter 611>

FIG. 56 is a block diagram depicting a configuration example of theclassification adaptive filter 611 of FIG. 55.

Referring to FIG. 56, the classification adaptive filter 611 includes animage conversion apparatus 631.

To the image conversion apparatus 631, a decoding in-progress image issupplied from the arithmetic operation section 205 (FIG. 55) andparameter information, a seed coefficient for each class as reductionfilter information and encoding information are supplied from thereversible decoding section 202.

The image conversion apparatus 631 calculates tap coefficients for eachclass from the parameter z obtained from the parameter information andthe seed coefficient for each class as the reduction filter informationsimilarly to the image conversion apparatus 532 of FIG. 44, and performsimage conversion by a classification adaptive process using the tapcoefficients for each class using the decoding in-progress image as afirst image to convert the decoding in-progress image as the first imageinto an after-filter image as a second image equivalent to an originalimage (generate an after-filter image) and supplies the after-filterimage to the sorting buffer 207 and the frame memory 210 (FIG. 55).

It is to be noted that the image conversion apparatus 631 performs, inthe classification adaptive process, classification using encodinginformation as occasion demands similarly to the image conversionapparatus 532 of FIG. 44.

<Configuration Example of Image Conversion Apparatus 631>

FIG. 57 is a block diagram depicting a configuration example of theimage conversion apparatus 631 of FIG. 56.

It is to be noted that, in FIG. 57, elements common to those of theimage conversion apparatus 231 of FIG. 22 are denoted by like referencenumerals and description of them is hereinafter omitted suitably.

Referring to FIG. 57, the image conversion apparatus 631 includes thecomponents from the tap selection section 241 to the classificationsection 243, the prediction arithmetic operation section 245 and acoefficient acquisition section 641.

Accordingly, the image conversion apparatus 631 of FIG. 57 is common tothe image conversion apparatus 231 of FIG. 22 in that it includes thecomponents from the tap selection section 241 to the classificationsection 243 and the prediction arithmetic operation section 245.

However, the image conversion apparatus 631 of FIG. 57 is different fromthe image conversion apparatus 231 of FIG. 22 in that it includes thecoefficient acquisition section 641 in place of the coefficientacquisition section 244.

To the coefficient acquisition section 641, parameter information andseed information for each class as reduction filter information aresupplied from the reversible decoding section 202 (FIG. 55).

The coefficient acquisition section 641 generates tap coefficients foreach class similarly to the coefficient acquisition section 562 of FIG.50.

In particular, the coefficient acquisition section 641 generates tapcoefficients for each class in accordance with the expression (9) using,for example, the seed coefficient for each class as the reduction filterinformation and the parameter z obtained from the parameter information.Then, the coefficient acquisition section 641 acquires the tapcoefficients of the class of a noticed pixel supplied from theclassification section 243 from among the tap coefficients for theindividual classes and supplies the acquired tap coefficients to theprediction arithmetic operation section 245.

FIG. 58 is a block diagram depicting a configuration example of thecoefficient acquisition section 641 of FIG. 57.

Referring to FIG. 58, the coefficient acquisition section 641 includes astorage section 671, a tap coefficient calculation section 672, anotherstorage section 673 and an acquisition section 674.

To the storage section 671, a seed coefficient for each class as thereduction filter information is supplied from the reversible decodingsection 202 (FIG. 55).

The storage section 671 stores the seed coefficients for the individualclasses as the reduction filter information from the reversible decodingsection 202.

To the tap coefficient calculation section 672, parameter information issupplied from the reversible decoding section 202. The tap coefficientcalculation section 672 acquires a parameter z used in the encodingapparatus 11 (parameter z generated by the parameter generation section561 of FIG. 50) from the parameter information, and uses the parameter zand the seed coefficients for the individual classes stored in thestorage section 671 to calculate tap coefficients for each class inaccordance with the expression (9) and supplies the tap coefficients tothe storage section 673.

The storage section 673 stores the tap coefficients for each class fromthe tap coefficient calculation section 672.

To the acquisition section 674, a class of a noticed pixel is suppliedfrom the classification section 243.

The acquisition section 674 acquires the tap coefficients of the classof the noticed pixel from the classification section 243 from among thetap coefficients for the individual classes stored in the storagesection 673 and supplies the acquired tap coefficients to the predictionarithmetic operation section 245.

<Decoding Process>

FIG. 59 is a flow chart illustrating an example of the decoding processof the decoding apparatus 12 of FIG. 55.

In the decoding process, at step S271, the accumulation buffer 201temporarily accumulates encoded data transmitted from the encodingapparatus 11 similarly as at step S71 of FIG. 24 and suitably suppliesthe encoded data to the reversible decoding section 202, and thereafter,the processing advances to step S272.

At step S272, the reversible decoding section 202 receives and decodesthe encoded data supplied from the accumulation buffer 201 similarly asat step S72 of FIG. 24 and supplies quantization coefficients obtainedby the decoding to the dequantization section 203.

Further, in the case where encoding information, reduction filterinformation and parameter information are obtained by decoding of theencoded data, the reversible decoding section 202 supplies necessaryencoding information to the intra-prediction section 212, motionprediction compensation section 213 and other necessary blocks.

Further, the reversible decoding section 202 supplies the encodinginformation, reduction filter information and parameter information tothe classification adaptive filter 611.

Thereafter, the processing advances from step S272 to step S273, atwhich the classification adaptive filter 611 decides whether reductionfile information is supplied from the reversible decoding section 202.

In the case where it is decided at step S273 that reduction filterinformation is not supplied, the processing advances to step S275skipping step S274.

On the other hand, in the case where it is decided at step S273 thatreduction filter information is supplied, the processing advances tostep S274, at which the coefficient acquisition section 641 thatconfigures the image conversion apparatus 631 (FIG. 57) of theclassification adaptive filter 611 acquires a seed coefficient for eachclass as the reduction filter information.

Further, in the coefficient acquisition section 641 (FIG. 58), thestorage section 671 updates the seed coefficients for the individualclasses stored in the storage section 671 to the seed coefficients forthe individual classes as the reduction filter information (stores theseed coefficients for the individual classes as the reduction filterinformation in an overwriting state).

Then, the processing advances from step S274 to step S275, at which aprediction decoding processing is performed, and the decoding processends.

FIG. 60 is a flow chart illustrating an example of the predictiondecoding process at step S275 of FIG. 59.

In the prediction decoding process, processes similar to those at stepsS81 to S89 of FIG. 25 are performed at steps S281 to S289, respectively.

It is to be noted that, although, at step S286, the classificationadaptive filter 611 performs a classification adaptive process as aprocess of an ILF for a decoding in-progress image from the arithmeticoperation section 205 similarly as at step S86 of FIG. 25, in theclassification adaptive process, tap coefficients for the individualclasses generated from the seed coefficients for the individual classesas the reduction filter information acquired by the coefficientacquisition section 641 at step S274 of FIG. 59 are used.

FIG. 61 is a flow chart illustrating an example of the classificationadaptive process performed at step S286 of FIG. 60.

In the image conversion apparatus 631 (FIG. 57) of the classificationadaptive filter 611, at step S291, the tap selection section 241 selectsone of pixels that have not been selected as a notice pixel as yet as anoticed pixel from within the decoding in-progress image supplied fromthe arithmetic operation section 205 similarly as at step S91 of FIG.26, and the processing advances to step S292.

At step S292, the coefficient acquisition section 641 of the imageconversion apparatus 631 (FIG. 57) decides whether parameter informationregarding a noticed image is supplied from the reversible decodingsection 202 (FIG. 55).

In the case where it is decided at step S292 that parameter informationregarding a noticed pixel is not supplied, the processing advances tostep S294 skipping step S293.

In particular, in the case where the parameter information regarding thenoticed pixel is same as parameter information of the pixel that havebeen selected as the noticed pixel in the preceding operation cycle andno parameter information is transmitted from the encoding apparatus 11,the processing advances from step S292 to step S294 skipping step S293.Thereafter, prediction arithmetic operation of the expression (1) fordetermining an after-filter image is performed using tap coefficientscalculated using a parameter z obtained from parameter information mostrecently supplied to the classification adaptive filter 611 from thereversible decoding section 202 (FIG. 55).

On the other hand, in the case where it is decided at step S292 thatparameter information regarding a noticed pixel is supplied from thereversible decoding section 202 (FIG. 55), the processing advances tostep S293, at which new tap coefficients are calculated using aparameter z obtained from the parameter information.

In particular, at step S293, the tap coefficient calculation section 672of the coefficient acquisition section 641 (FIG. 58) calculates tapcoefficients for the individual classes using the seed coefficients forthe individual classes as reduction filter information stored in thestorage section 671 at step S274 of FIG. 59 and the parameter z obtainedfrom the parameter information regarding the noticed pixel and suppliedfrom the reversible decoding section 202 (FIG. 55).

Further, the tap coefficient calculation section 672 supplies the tapcoefficients for the individual classes to the storage section 673 so asto be stored in an overwriting relationship, and the processing advancesfrom step S293 to step S294.

At step S294, the tap selection sections 241 and 242 select a predictiontap and a class tap regarding the noticed pixel from within the decodingin-progress image supplied from the arithmetic operation section 205similarly as at step S92 of FIG. 26 and supplies them to the predictionarithmetic operation section 245 and the classification section 243,respectively.

Thereafter, the processing advances from step S294 to step S295, atwhich the classification section 243 performs classification of thenoticed pixel using the class taps regarding the noticed pixel and theencoding information regarding the noticed pixel similarly as at stepS93 of FIG. 26.

Then, the classification section 243 supplies the class of the noticepixel obtained by the classification of the noticed pixel to thecoefficient acquisition section 641, and the processing advances fromstep S295 to step S296.

At step S296, the acquisition section 674 of the coefficient acquisitionsection 641 (FIG. 58) acquires tap coefficients of the class of thenoticed pixel from the classification section 243 from among the tapcoefficients for the individual classes stored in the storage section673 at step S293 immediately previously and supplies the acquired tapcoefficients to the prediction arithmetic operation section 245, and theprocessing advances to step S297.

At steps S297 to S299, processes similar to those at steps S95 to S97 ofFIG. 26 are performed, respectively.

As described above, in the encoding apparatus 11 of FIG. 43 and thedecoding apparatus 12 of FIG. 55, since an ILF process is performed by aclassification adaptive process, an after-filter image closer to anoriginal image than a processing result of the ILF can be obtained. As aresult, the S/N of the decoded image can be improved. Further, since anafter-filter image closer to the original image can be obtained, theresidual becomes small, and therefore, the compression efficiency can beimproved. Further, in the encoding apparatus 11, since an order for aseed coefficient for each class is selected such that the seedcoefficient evaluation value becomes best to generate a seed coefficientfor each class as reduction filter information that reduces tapcoefficients for each class and not the tap coefficients for theindividual classes but the reduction filter information is transmittedto the decoding apparatus 12, the compression efficiency can be improvedfurther.

<Different Example of Reduction Filter Information that Reduces TapCoefficients>

FIG. 62 is a view illustrating a different example of reduction filterinformation that reduces tap coefficients for each class obtained by tapcoefficient learning.

As the reduction filter information that reduces tap coefficients,arbitrary information that can be obtained by reducing tap coefficientsfor each class obtained by tap coefficient learning can be adopted inaddition to tap coefficients of a class whose merit decision value isequal to or higher than a threshold value, tap coefficients and acorresponding relationship LUT for each integration class, and seedcoefficients for each class.

Here, since the tap coefficients become an overhead of encoded data, asthe data amount of the tap coefficients increases, the compressionefficiency decreases. Further, if the number of tap coefficients of oneclass (N of the expression (1)) increases, the arithmetic operation costfor prediction arithmetic operation of the expression (1) increases.

By reducing tap coefficients for each class obtained by tap coefficientlearning, it is possible to suppress decrease of the compressionefficiency and increase of the arithmetic operation cost for predictionarithmetic operation.

Reduction of tap coefficients for each class obtained by tap coefficientlearning can be performed, for example, by regeneration of classes ofreducing the total number of classes or reduction of tap coefficientsthemselves.

For example, in the case where a class tap is configured from ninepixels of a cross shape centered at a noticed pixel as depicted in FIG.62 and classification by a 1-bit ADRC process is performed, for example,by inverting the bits of an ADRC code whose most significant bit is 1,the class number can be degenerated from 512=2⁹ classes to 256=2⁸classes. With 256 classes after degeneration of classes, the data amountof the tap coefficients is reduced to ½ in comparison with that in analternative case in which an ADRC code (of a 1-bit ADR process) of aclass tap of nine pixels is used as it is as a class code.

Further, by performing degeneration of classes for integrating classesbetween which ADRC results of pixels that are in a line symmetricalpositional relationship in an upward and downward direction, in aleftward and rightward direction or in an oblique direction from amongthe cross-shaped nine pixels configuring the class tap are same as eachother into one class, the class number can be reduced to 100 classes. Inthis case, the data amount of tap coefficients of 100 classes isapproximately 39% of the data amount of tap coefficients of 256 classes.

Further, by performing, in addition to the foregoing, degeneration ofclasses of collecting classes between which ADRC results of pixels thatare in a point symmetrical positional relationship from among thecross-shaped nine pixels configuring the class tap are same as eachother into one class, the class number can be reduced to 55 classes. Inthis case, the data amount of tap coefficients of 55 classes isapproximately 21% of the data amount of tap coefficients of 256 classes.

Reduction of tap coefficients not only can be performed by degenerationof classes as described above but also can be performed by reduction oftap coefficients themselves.

In particular, for example, in the case where a prediction tap and ablock (encoding block) are configured from same pixels, tap coefficientsthemselves can be reduced on the basis of a block phase.

For example, in the case where a prediction tap and a block areconfigured from 4×4 pixels, as tap coefficients of right upper 2×2pixels having a positionally line symmetrical relationship in theleft-right direction with left upper 2×2 pixels of the prediction tap,left lower 2×2 pixels having a positionally line symmetricalrelationship in the up-down direction and right lower 2×2 pixels havinga positional relationship of point symmetry, tap coefficients obtainedby re-arranging the tap coefficients of the left upper 2×2 pixels inaccordance with the positional relationship can be adopted. In thiscase, the 16 tap coefficients for the 4×4 pixels configuring theprediction tap can be reduced to four tap coefficients for the leftupper 2×2 pixels.

Further, it is possible to adopt, as tap coefficients of 4×2 pixels in alower half of the prediction tap having a positionally line symmetricalrelationship in the up-down direction with 4×2 pixels in an upper halfof the prediction tap, tap coefficients obtained by re-arranging the tapcoefficients of the 4×2 pixels in the upper half in accordance with thepositional relationship can be adopted. In this case, the 16 tapcoefficients for the 4×4 pixels configuring the prediction tap can bereduced to eight tap coefficients for the 4×2 pixels in the upper half.

Further, by adopting same tap coefficients as tap coefficients forpixels having a positionally line symmetrical relationship in theleft-right direction of the prediction tap or tap coefficients forpixels having a positionally line symmetrical relationship in an obliquedirection, tap coefficients can be reduced.

It is to be noted that, if such reduction of tap coefficients based on ablock phase as described above is performed bluntly, then the S/N of anafter-filter image obtained with the tap coefficients after thereduction degrades (errors with respect to an original image increase).

Therefore, reduction of tap coefficients based on a block phase isperformed such that, for example, a waveform pattern of pixelsconfiguring a prediction tap is analyzed depending upon an ADRC codeused in classification and, in the case were the waveform pattern hasspatial symmetry, a same tap coefficient can be adopted for pixels of aprediction tap having a positionally symmetrical relationship.

In particular, for example, in the case where a class tap is configuredfrom 2×2 pixels and an ADRC code where ADRC results of the pixels of theclass tap are lined up in a raster scan order is 1001, considering thatthe class tap has a positional relationship of point symmetry, a sametap coefficient can be adopted for tap coefficients of pixels of thepixel tap that has the positional relationship of line symmetry.

As described above, various methods can be adopted as the tapcoefficient reduction method for reducing tap coefficients.

In particular, as described hereinabove with reference to FIG. 62, aclass code utilization method of reducing (the number of) classes toreduce tap coefficients by operating a class code obtained for a classtap such as, for example, to perform inversion of bits of an ADRC codewhose most significant bit is 1, which is obtained for a class tap, todegenerate the class number to ½ or to integrate classes, in which ADRCresults of pixels having a positional relationship of line symmetry orpoint symmetry in a class tap are same as each other, into one class canbe adopted as the tap coefficient reduction method.

Further, a positional relationship utilization method of reducing tapcoefficients of each class by adopting, for pixels configuring aprediction tap, (tap coefficients same as) tap coefficients for pixelshaving a predetermined positional relationship such as line symmetry,point symmetry or the like with the pixels as described hereinabove, forexample, with reference to FIG. 62 can be adopted as the tap coefficientreduction method.

Furthermore, a coefficient selection method of reducing tap coefficientsby selecting, from among the latest coefficients determined by thelatest tap coefficient learning, only the latest coefficients of a classwith regard to which the merit decision value is equal to or higher thana threshold value as described hereinabove, for example, in connectionwith the first configuration example of the encoding apparatus 11 andthe decoding apparatus 12 of FIGS. 9 to 26 can be adopted as the tapcoefficient reduction method.

Further, an inter-coefficient distance utilization method of reducingtap coefficients by integrating a plurality of classes such as twoclasses of a combination in which the inter-coefficient distance issmallest such that the tap coefficient evaluation value (RD cost or thelike) as described hereinabove is improved, for example, in connectionwith the second configuration example of the encoding apparatus 11 andthe decoding apparatus 12 of FIGS. 27 to 42 can be adopted as the tapcoefficient reduction method.

Furthermore, a seed coefficient utilization method of reducing tapcoefficients by determining a seed coefficient that defines arelationship curve that fits with tap coefficients for a plurality ofvalues of the parameter z as described hereinabove, for example, inconnection with the third configuration example of the encodingapparatus 11 and the decoding apparatus 12 of FIGS. 43 to 61 can beadopted as the tap coefficient reduction method.

It is to be noted that the tap coefficient reduction method is notlimited to any of the class code utilization method, positionalrelationship utilization method, coefficient selection method,inter-coefficient distance utilization method and seed coefficientutilization method described above, but an arbitrary method that reducestap coefficients can be adopted.

In particular, as the tap coefficient reduction method, for example, amethod for compressing tap coefficients by a reversible compressionmethod, a class evaluation value utilization method for reducing tapcoefficients by deleting, on the basis of a class evaluation value forevaluating a performance of tap coefficients for each class, a classother than a class whose class performance evaluation value is equal toor higher than a threshold value and so forth are additionallyavailable.

Here, as a class evaluation value that evaluates a performance of tapcoefficients of a certain noticed class, for example, an RD cost in thecase where encoding is performed using an after-filter image obtained byperforming predictive arithmetic operation using tap coefficients of thenoticed class, a PSNR (Peak signal-to-noise ratio), AMSE that is thedifference (first MSE-second MSE) between a first MSE (Mean SquaredError) between an image and an original image obtained by performing afilter process of a general ILF that does not use a classificationadaptive process and a second MSE between an after-filter image and anoriginal image obtained by performing prediction arithmetic operationusing tap coefficients of a noticed class, and so forth can be adopted.

FIG. 63 is a view illustrating an example of reduction of tapcoefficients by the class evaluation value utilization method.

FIG. 63 depicts an example of a class evaluation value (of tapcoefficients) of initial classes obtained by tap coefficient learningwhen the initial classes are lined up in an ascending order of AMSE as aclass evaluation value of tap coefficients for each initial class.

Referring to FIG. 63, the axis of abscissa represents initial classeslined up in the ascending order of the class evaluation value, and theaxis of ordinate represents class evaluation values of the initialclasses.

Here, AMSE as the class evaluation value of a certain initial classrepresents a picture quality improvement effect regarding by what degreethe picture quality of an after-filter image obtained by predictionarithmetic operation using tap coefficients of the initial class isimproved in comparison with the picture quality of an image obtained byperforming a filter process of an ILF that does not use theclassification adaptive process.

In reduction of tap coefficients by the class evaluation valueutilization method, one initial class from among initial classes likedup in an ascending order of the class evaluation value is set as aboundary class. Then, an initial class whose rank of the classevaluation value is lower than that of the boundary class is deleted asa class of a target of deletion.

As the boundary class, for example, an initial class whose classevaluation value is in the minimum from among initial classes whoseclass evaluation value is equal to or higher than a threshold value oran initial class whose class evaluation value is in the maximum fromamong initial classes whose class evaluation value is equal to or lowerthan the threshold value, an initial value whose rank of the classevaluation value is a rank determined in advance, and so forth can beadopted.

<Fourth Configuration Example of Encoding Apparatus 11>

FIG. 64 is a block diagram depicting a fourth configuration example ofthe encoding apparatus 11 of FIG. 1.

It is to be noted that, in FIG. 64, elements corresponding to those ofFIG. 9 are denoted by like reference numerals and description of them ishereinafter omitted suitably.

Referring to FIG. 64, the encoding apparatus 11 includes the componentsfrom the A/D conversion section 101 to the arithmetic operation section110 and from the frame memory 112 to the rate controlling section 117and a classification adaptive filter 911.

Accordingly, the encoding apparatus 11 of FIG. 64 is common to that ofFIG. 9 in that it includes the components from the A/D conversionsection 101 to the arithmetic operation section 110 and from the framememory 112 to the rate controlling section 117.

However, the encoding apparatus 11 of FIG. 64 is different from that ofFIG. 9 in that it includes the classification adaptive filter 911 inplace of the classification adaptive filter 111.

The classification adaptive filter 911 is a filter that functions as anILF by performing a classification adaptive process and is common to theclassification adaptive filter 111 in that it performs an ILF process bya classification adaptive process.

However, the classification adaptive filter 911 is different from theclassification adaptive filter 111 in that, in the classificationadaptive filter 911, since tap coefficients of a plurality of initialclasses determined by tap coefficient learning are reduced by areduction process, tap coefficients degenerated to a data amount smallerthan that of the tap coefficients of the plurality of initial classesare generated as reduction filter information.

Further, the classification adaptive filter 911 is different from theclassification adaptive filter 111 in that, while, in the classificationadaptive filter 911, degeneration for converting tap coefficients ofinitial classes into tap coefficients of a smaller data amount isperformed by a degeneration method selected from among a plurality ofdegeneration methods, degeneration information representative of thedegeneration method is generated as reduction filter information.

Furthermore, the classification adaptive filter 911 is different fromthe classification adaptive filter 111 in that it performs aclassification adaptive process using tap coefficients afterdegeneration as reduction filter information and degenerationinformation.

<Configuration Example of Classification Adaptive Filter 911>

FIG. 65 is a block diagram depicting a configuration example of theclassification adaptive filter 911 of FIG. 64.

Referring to FIG. 65, the classification adaptive filter 911 includes alearning apparatus 931, a reduction apparatus (reduction section) 932,and an image conversion apparatus 933.

To the learning apparatus 931, an original image is supplied from thesorting buffer 102 (FIG. 64) and a decoding in-progress image issupplied from the arithmetic operation section 110 (FIG. 64).Furthermore, encoding information is supplied to the learning apparatus931.

The learning apparatus 931 performs tap coefficient learning fordetermining tap coefficients for each initial class using the decodingin-progress image as student data and using the original data as teacherdata.

Further, the learning apparatus 931 supplies tap coefficients for theindividual initial classes obtained by the tap coefficient learning tothe reduction apparatus 932.

It is to be noted that the learning apparatus 931 performsclassification in tap coefficient learning using encoding information asoccasion demands.

Further, the learning apparatus 931 performs, in the tap coefficientlearning, classification of a noticed pixel using a plurality of kindsof pixel-related information relating to the noticed pixel.

The pixel-related information is roughly divided into an imagecharacteristic amount such as an ADRC code or the like and encodinginformation such as a quantization parameter QP or the like. As theplurality of kinds of pixel-related information, a combination of one ormore kinds of image characteristic amounts and one or more kinds ofencoding information, a combination of a plurality of kinds of imagecharacteristic amounts and 0 or more kinds of encoding information and acombination of 0 or more kinds of image characteristic amounts and aplurality of kinds of encoding information can be adopted.

To the reduction apparatus 932, tap coefficients of initial classesobtained by tap coefficient learning are supplied from the learningapparatus 931 as described hereinabove. Further, to the reductionapparatus 932, an original image that becomes teacher data, a decodingin-progress image that becomes student data and encoding information aresupplied similarly to the learning apparatus 931.

The reduction apparatus 932 performs a reduction process fordegenerating tap coefficients for individual initial classes from thelearning apparatus 931 to tap coefficients of a data amount smaller thanthat of the tap coefficients.

It is to be noted that the degeneration of tap coefficients is roughlydivided into degeneration that decreases tap coefficients themselves ofeach class and degeneration of classes for reducing classes. Althoughthe reduction apparatus 932 can perform one or both of the degenerationof reducing tap coefficients themselves of each class and thedegeneration of classes for reducing classes, in the following,description is given assuming that at least degeneration of classes isperformed as the degeneration of tap coefficients by the reductionapparatus 932.

The reduction apparatus 932 performs degeneration of tap coefficientsfor individual initial classes and determines tap coefficients forindividual degeneration classes of a class number smaller than the classnumber of initial classes from the tap coefficients for the individualinitial classes by the degeneration.

The reduction apparatus 932 performs degeneration for obtaining tapcoefficients for individual degeneration classes from tap coefficientsfor individual initial classes by a degeneration method selected fromamong a plurality of degeneration methods.

In the reduction apparatus 932, selection of a degeneration method to beused to obtain tap coefficients for individual degeneration classes isperformed using an original image as teacher data, a decodingin-progress image as student data and encoding information.

The reduction apparatus 932 generates tap coefficients for individualdegeneration classes obtained by degeneration in the reduction methodand degeneration information representative of a degeneration methodused to obtain the tap coefficients for the individual degenerationclasses as reduction filter information and supplies the reductionfilter information to the image conversion apparatus 933 and thereversible encoding section 106 (FIG. 64).

To the image conversion apparatus 933, a decoding in-progress image issupplied from the arithmetic operation section 110 (FIG. 64) andreduction filter information is supplied from the reduction apparatus932. Further, encoding information is supplied to the image conversionapparatus 933.

The image conversion apparatus 933 performs, using the decodingin-progress image as a first image, image conversion by a classificationadaptive process in which tap coefficients for the individualdegeneration classes and the degeneration information included in thereduction filter information from the reduction apparatus 932 to convertthe decoding in-progress image as the first image into an after-filterimage as a second image equivalent g to the original image (to generatean after-filter image), and supplies the after-filter image to the framememory 112 (FIG. 64).

It is to be noted that the image conversion apparatus 933 performsclassification in the classification adaptive process using the encodinginformation as occasion demands.

<Configuration Example of Learning Apparatus 931>

FIG. 66 is a block diagram depicting a configuration example of thelearning apparatus 931 of FIG. 65.

Referring to FIG. 66, the learning apparatus 931 includes a learningsection 940 and a storage section 946. The learning section 940 includestap selection sections 941 and 942, a classification section 943, anaddition section 944 and a coefficient calculation section 945.

The learning section 940 is common to the learning section 33 of FIG. 13in that it determines tap coefficients for each class by tap coefficientlearning.

However, the learning section 940 is different from the learning section33 of FIG. 13 in that classification is performed substantially byvarious classification methods and tap coefficients for several thousandto several ten thousand individual initial classes are determined while,in the learning section 33, tap coefficients of the number of classessufficiently smaller than such a huge number as mentioned above aredetermined.

The tap selection section 941 successively selects pixels configuring adecoding in-progress image as student data as a noticed pixel similarlyto the tap selection section 41 of FIG. 13 and supplies informationrepresentative of the noticed pixel to necessary blocks.

Further, the tap selection section 941 selects, in regard to the noticedpixel, one or more pixels that become a prediction tap from among pixelsthat configure a decoding in-progress image as student data and suppliesthe prediction tap configured from one or more pixels to the additionsection 44.

The tap selection section 942 selects, in regard to the noticed pixel,one or more pixels that become a class tap from among pixels thatconfigure a decoding in-progress image as student data and supplies theclass tap configured from one or more pixels to the classificationsection 943.

It is to be noted that the tap selection section 942 configures, inregard to the noticed pixel, a plurality of class taps to be used todetect image characteristic amounts as a plurality of kinds ofpixel-related information.

The classification section 943 performs, in regard to the noticed pixel,classification using image characteristic amounts and encodinginformation detected from a plurality of class taps from the tapselection section 942, namely, using a plurality of kinds ofpixel-related information, and outputs a class code corresponding to aclass of the noticed pixel obtained as a result of the classification tothe addition section 944.

Since the classification section 943 performs classification using aplurality of kinds of pixel-related information in such a manner asdescribed above, a noticed pixel is classified to one of, for example,several thousand to several ten thousand or more classes.

Classes obtained by the classification section 943 are initial classes,and accordingly, the class number of initial classes is equal to a classnumber of classes that can be classified by the classification section943.

The addition section 944 acquires, similarly to the addition section 44of FIG. 13, (a pixel value of) a corresponding pixel corresponding tothe noticed pixel from among pixels configuring the original image asteacher data and performs addition of the corresponding pixel and (pixelvalues of) pixels in the decoding in-progress image as student dataconfiguring a prediction tap of the noticed pixel supplied from the tapselection section 941 for each (of class codes) of the initial codessupplied from the classification section 943.

Then, the addition section 944 sets up the normal equation indicated bythe expression (8) for each initial class by performing the addition andsupplies the normal equations to the coefficient calculation section945.

The coefficient calculation section 945 solves the normal equation foreach initial class supplied from the addition section 944 to determinetap coefficients (w_(n)) for each of the class number of initial classesthat can be classified by the classification section 943 similarly tothe coefficient calculation section 45 of FIG. 13, and stores the tancoefficients (w_(n)) into the storage section 946.

The storage section 946 stores the tap coefficients for the individualinitial classes from the coefficient calculation section 945.

FIG. 67 is a block diagram depicting a configuration example of the tapselection section 42 and the classification section 943 of FIG. 66.

Referring to FIG. 67, the tap selection section 942 includes one or moreselection sections 951 ₁, 951 ₂, . . . , 951 _(H-1). Meanwhile, theclassification section 943 includes a plurality of, H, informationdetection sections 952 ₁, 952 ₂, . . . , 952 _(H), a plurality of, H,subclass classification sections 953 ₁, 953 ₂, . . . , 953 _(E) and aclass configuration section 954.

To a selection section 951 _(h), a decoding in-progress image as studentdata is supplied. The selection section 951 selects pixels, which becomean hth class tap to be used for detection of the hth imagecharacteristic amount from within the decoding in-progress image andconfigures the hth class tap from the pixels, and supplies the hth classtap to the information detection section 952 _(h).

Here, the hth class tap configured from a certain selection section 951_(h) and the h′th class tap configured from a different selectionsection 951 _(h′) may have a same tap structure or may have tapstructures different from each other.

The information detection section 952 _(h) detects an hth imagecharacteristic amount from pixels configuring the hth class tap from theselection section 951 _(h) and supplies the hth image characteristicamount as hth pixel-related information to the subclass classificationsection 953 _(h).

The information detection section 952 _(h) can detect an imagecharacteristic amount such as, for example, an ADRC code obtained by an1-bit ADRC process of the hth class tap, a DR (Dynamic Range) of pixelvalues of pixels configuring the hth class tap or the like.

It is to be noted that, to the information detection section 952 _(H)from among the information detection sections 952 ₁ to 952 _(H),encoding information of a noticed pixel is supplied. The informationdetection section 952 _(H) detects necessary information from theencoding information of the notice pixel, and processes the informationfurther as occasion demands, and then supplies the processed informationas Hth pixel-related information to the subclass classification section953 _(H).

The subclass classification section 953 _(h) subclass-classifies anoticed pixel to an hth subclass as a subset of an (final) initial classof the noticed pixel using the hth image-related information suppliedfrom the information detection section 952 _(h) and supplies the hthsubclass of the noticed pixel obtained as a result of the subclassclassification to the class configuration section 954.

As a method for the subclass classification by the subclassclassification section 953 _(h), for example, a method of performing athreshold process of comparing the hth pixel-related information withone or a plurality of threshold values and outputting a valuerepresentative of a result of the comparison between the hthpixel-related information and the threshold value or values as asubclass (code), a method of outputting, in the case where the hthpixel-related information is a bit string such as, for example, an ADRCcode, the bit string as it is as a subclass, and so forth are available.

It is to be noted that, in the case where a threshold value process ofpixel-correlated information is performed to perform subclassclassification, even if pixel-related information is pixel-relatedinformation of a same type, subclasses that are different from eachother in threshold value to be used for a threshold value process of thepixel-related information are subclasses different (in method) from eachother.

Meanwhile, in the case where the hth class tap and the h′th class taphave tap structures different from each other, even if the informationdetection sections 952 _(h) and 952 _(h′) detect pixel-relatedinformation of a same type from the hth class tap and the h′th classtap, respectively, the subclass that uses the pixel-related informationdetected from the hth class tap and the subclass that uses thepixel-related information detected from the h′th class tap aresubclasses of different methods from each other.

The class configuration section 954 configures an initial class of anoticed pixel from a combination of H subclasses from the first to theHth subclasses supplied from the subclass classification section 953 ₁to 953 _(H), and supplies the initial class to the addition section 944(FIG. 66).

The classification performed by the classification section 943 isclassification of various methods depending upon the tap structure of aclass tap configured by the selection section 951 _(h), the type ofimage-related information detected by the information detection section952 _(h) and a method of subclass classification (subclassclassification method) performed by the subclass classification section953 _(h).

Further, it can be considered that the classification performed by theclassification section 943 includes classification of various methods(classification methods).

It is to be noted that, while, in FIG. 67, the number of informationdetection sections 952 _(h) and the number of subclass classificationsections 953 _(h) are made equal to each other in order to simplifydescription, the number of information detection sections 952 _(h) andthe number of subclass classification sections 953 _(h) may not be equalto each other.

Further, (hth) pixel-related information detected by a certaininformation detection section 952 _(h) is supplied to the subclassclassification section 953 _(h) or is not supplied to the subclassclassification sections 953 _(h) but supplied to a different subclassclassification sections 953 _(h′).

Furthermore, the subclass classification sections 953 _(h′) performssubclass classification using (the h′th) pixel-related informationsupplied from the information detection sections 952 _(h′) or canperform subclass classification using pixel-related information suppliedfrom the information detection sections 952 _(h) or using both ofpixel-related information supplied from the information detectionsections 952 _(h) and 952 _(h′).

Further, in the classification section 943, the class configurationsection 954 not only can perform classification for configuring aninitial class using a combination of H subclasses supplied from thesubclass classification sections 953 ₁ to 953 _(H) but also can performclassification for configuring an initial class using a combination ofH1, H2, . . . subclasses smaller than H from among the H subclasses.

It can be considered that classification for configuring an initialclass using a certain combination from among combinations of one or aplurality of subclasses equal to or smaller than H and classificationfor configuring an initial class using a different combination areclassification of different methods from each other.

The classification section 943 not only can perform classification ofone method but also can perform classification of a plurality of methods(a plurality of kinds of classification methods).

Thus, the classification section 943 can select, in the case whereclassification is performed by a plurality of kinds of classificationmethods, a classification method by which the picture quality of adecoded image and the encoding efficiency are best among the pluralityof kinds of classification methods, namely, a classification method thatis best, for example, in RD cost or the k, as an optimum classificationmethod. Further, the classification section 943 can output an initialclass obtained by the classification performed by the optimumclassification method as the class of the noticed pixel. Further, theencoding apparatus 11 can place and transmit information representativeof the optimum classification method into and together with reductionfilter information to the decoding apparatus 12, and the decodingapparatus 12 can perform classification by the optimum classificationmethod.

<Example of Image Characteristic Amount that Becomes Pixel-RelatedInformation>

FIG. 68 is a view depicting examples of an image characteristic amountthat becomes pixel-related information.

As the image characteristic amount that becomes pixel-relatedinformation, for example, an ADRC code, a DR, DiffMax, constancy,activity, a secondary differential sum, a maximum direction difference,a filter bank output and so forth can be adopted.

The ADRC code can be determined in such a manner as describedhereinabove with reference to FIG. 2 and so forth. In particular, forexample, a 1-bit ADRC code can be determined by dividing a pixel valuesuch as a luminance or the like of a pixel configuring a class into twovalues with a threshold value and lining up such pixel values as binaryvalues.

According to the (subclass) classification using an ADRC code, awaveform pattern (edge, texture (including a direction) and so forth) of(a pixel group configuring) a class tap is classified exhaustively, andin a classification adaptive process, an optimum restoration effect ofan image can be obtained for each waveform pattern of a class tap in aclassification adaptive process.

The DR is a difference between a maximum value and a minimum value of apixel value such as the luminance or the like of pixels configuring theclass tap. In the case where the DR is low, the classification that usesthe DR contributes to reduction of noise and so forth of a flattenedportion, and in the case where the DR is high, the classificationcontributes to restoration of an edge.

DiffMax is a maximum value of difference absolute values of pixel valuesof pixels adjacent each other in horizontal, vertical and obliquedirections in a class tap. Classification that uses DiffMax contributes,in the case where DiffMax is small in a classification adaptive process,to reduction of a false contour of gradation, but contributes, in thecase where DiffMax is great, to restoration of a steep edge (offset).

It is to be noted that the combination of DR and DiffMax, in particular,for example, DiffMax/DR or a biaxial direction of DiffMax and DR(DiffMax, DR) can be made, as an image characteristic amount differentfrom mere DR or mere DiffMax, an index to what number of pixels arerequired to climb up the amplitude of the DR in a class tap.

The constancy can be represented by a value that represents, forexample, in a class tap, a difference between the difference absolutevalue sums in a direction in which the difference absolute value sum ofpixel values of pixels adjacent each other in each direction is in themaximum and in another direction in which the difference absolute valuesum is in the minimum. The classification that uses the constancycontributes, in the case where the constancy is small in aclassification adaptive process, to restoration of (a fine design suchas) a texture (or noise), but contributes, in the case where theconstancy is high, to restoration of an edge (structural line).

The activity can be represented, for example, by a difference absolutevalue sum of pixel values of pixels adjacent each other in thehorizontal and vertical directions in a class tap. The classificationthat uses the activity contributes, in the case where the activity issmall, to restoration of a step edge (simple pattern), but contributes,in the case where the activity is great, to restoration of a texture(complicated pattern).

The secondary differential sum is, for example, an absolute value sum ofsecondary differential of pixel values of pixels adjacent each other inthe horizontal and vertical directions in a class tap. Theclassification that uses the secondary differential sum contributes, inthe case where the secondary differential sum is small, to restorationof a step edge, but contributes, in the case where the secondarydifferential sum is great, to restoration of a texture.

The maximum direction difference is a value representative of adirection in which the difference absolute sum of pixel values of pixelsadjacent each other in the horizontal, vertical and oblique directionsin a class tap is in the maximum. The classification that uses themaximum direction difference classifies the direction of an amplitude, agradient, a structure or the like of a periphery of a noticed pixel, andconsequently, in a classification adaptive process, an optimumrestoration effect of an image can be obtained for each direction of theamplitude, gradient, structure or the like of the periphery of thenoticed pixel.

The filter bank output is a value obtained by inputting pixel values ofpixels configuring a class tap to a plurality of directional band passfilters. Although the classification that uses the filter bank outputrequires a high calculation cost, it is high in classification accuracyin comparison with classification that uses the maximum directiondifference.

As the image characteristic amount to be detected by the informationdetection sections 952 _(h), an arbitrary image characteristic amountcan be adopted in addition to such an ADRC code, DR, DiffMax, constancy,activity, secondary differential sum, maximum direction difference andfilter bank output as described above.

<Example of Configuration of Class>

FIG. 69 is a view depicting an example of a method by which the classconfiguration section 954 of FIG. 67 configures an initial class fromfirst to Hth subclasses supplied from the subclass classificationsections 953 ₁ to 953 _(H).

A of FIG. 69 is a view illustrating a first method of configuring aninitial class from first to Hth subclasses.

The class configuration section 954 can determine the product of thefirst to Hth subclasses and can determine the product as an initialclass. In particular, the class configuration section 954 can configurea new bit string in which H bit strings representative of the first toHth subclasses are lined up, and can output the new bit string as a bitstring representative of the initial class.

B of FIG. 69 is a view illustrating a second method for configuring aninitial class from first to Hth subclasses.

The class configuration section 954 plots, in a subclass space(pixel-related information space) having axes of the first to Hthsubclasses (pixel-related information), for example, pointscorresponding to first to Hth subclasses (pixel-related information)obtained for individual pixels of a decoding in-progress image asstudent data from within a set (hereinafter referred to as learningsample) of an original image as teacher data and a decoding in-progressimage as student data to be used for tap coefficient learning by thelearning apparatus 931.

Further, the class configuration section 954 applies the k-means methodor the like to points corresponding to the first to Hth subclassesobtained for the learning samples plotted in the subclass space tocluster the subclass space into a plurality of clusters.

Then, the class configuration section 954 outputs a value correspondingto the cluster to which the points corresponding to the first to Hthsubclasses of the noticed pixel are clustered as an initial class of thenoticed pixel.

<Example of Combination of Plural Kinds of Pixel-Related Informationthat Uses Classification>

FIG. 70 is a view depicting a first example of a combination of aplurality of kinds of pixel-related information used for classificationof a notice pixel.

As the combination of a plurality of kinds of pixel-related informationto be used for classification of a noticed pixel, for example, thecombination of the ADRC code and the DR can be used.

According to subclass classification that uses the ADRC code, waveformpattern classification is performed by which a noticed pixel isclassified by a waveform pattern of a periphery of the noticed pixel.Meanwhile, according to subclass classification that uses the DR,amplitude classification is performed by which a notice pixel isclassified by the amplitude of a periphery of the noticed pixel.

As a result, according to the classification that uses the combinationof the ADRC code and the DR, namely, according to that classificationthat uses, for example, a result of subclass classification that usesthe ADRC code and a result of subclass classification that uses the DR,the noticed pixel is classified by an edge and a texture of a peripheryof the notice pixel.

FIG. 71 is a view depicting a second example of a combination of aplurality of kinds of pixel-related information to be used forclassification of a noticed pixel.

As the combination of a plurality of n of pixel-related information tobe used for classification of a noticed pixel, for example, acombination of the ADRC code, DR and DiffMax/DR can be used.

According to the subclass classification that uses DiffMax/DR, gradientclassification is performed by which a noticed pixel is classified bythe gradient of a periphery of the noticed pixel.

The combination of the ADRC code, DR and DiffMax/DR is a combination inwhich DiffMax/DR is added to the combination of the ADRC code and the DRof FIG. 70. Accordingly, according to that classification that uses theADRC code, DR and DiffMax/DR, a noticed pixel is classified by an edge,a gradation and a texture of a periphery of the noticed pixel.

Here, the ADRC code is a comparison result of a pixel value such as aluminance or the like of each pixel configuring a class tap and athreshold value and includes information of a direction of change of thepixel value of a periphery of the noticed pixel, and according to(subclass) classification that uses such an ADRC code as just described,a waveform pattern of a periphery of the noticed pixel can be classifiedexhaustively.

However, since the ADRC code does not include information of theamplitude or gradient of a waveform pattern of a periphery of a noticedpixel, with classification that uses the ADRC code, it is sometimesdifficult to classify with a higher degree of accuracy and reduce such afailure as noise of a flat portion or a false contour of a gradationportion or to distinguish a large edge and a small edge to performrestoration in accordance with the strength of the edge.

Thus, by combining the DR and DiffMax/DR with the ADRC code andperforming classification using the combination of the ADRC code, DR andDiffMax/DR, it is possible to perform restoration suitable for an imageof a local region of a periphery of the noticed pixel.

In particular, according to the classification that uses the combinationof the DR and DiffMax/DR with the ADRC code, waveform patterns of aperiphery of the noticed pixel can be classified exhaustively. Further,by the DR and DiffMax/DR, it is possible to classify with a high degreeof accuracy and reduce a failure like noise of a flat portion or a falsecontour of a gradation portion or to distinguish a great edge and asmall edge from each other to perform restoration in accordance with thestrength of the edge.

For example, in the case where both the DR and DiffMax/DR are small,false contours and so forth at flat portions or gradation portions canbe classified and reduced. Further, for example, in the case where theDR is small and DiffMax/DR is great, it is possible to classify andreduce noise such as block noise or the like at flat portions.Furthermore, for example, in the case where the DR is great andDiffMax/DR is small, it is possible to classify edges of gentlegradients such that such an edge may not be emphasized excessively.Further, for example, in the case where both the DR and DiffMax/DR aregreat, it is possible to classify steep edges and restore such edges inan appropriately emphasized state.

FIG. 72 depicts a third example of a combination of a plurality of kindsof pixel-related information to be used for classification of a noticedpixel.

As the combination of a plurality of kinds of pixel-related informationto be used for classification of a noticed pixel, for example, acombination of a set (combination) of the secondary differential sum andthe DR, a set of DiffMax and the DR and a maximum direction differencecan be used.

It is to be noted that, in regard to the set of the secondarydifferential sum and the DR, constancy or activity can be used in placeof the secondary differential sum. Further, in regard to the maximumdirection difference, the filter bank output can be used in place of themaximum direction difference.

According to the classification that uses a combination of a set(combination) of the secondary differential sum and the DR, a set ofDiffMax and the DR and a maximum direction difference, for example, anedge and a texture can be classified by the set of a secondarydifferential sum and a DR. Further, for example, by the set of DiffMaxand the DR (for example, DiffMax/DR), the gradient of an edge and soforth can be classified, and by the maximum direction difference, thedirection of an edge can be classified.

As a result, according to the combination of the set of the secondarydifferential sum and the DR, set of DiffMax and the DR and maximumdirection difference, an edge, a texture and a gradation (including adirection of the gradation) can be classified.

<Configuration Example of Deletion Apparatus 932>

FIG. 73 is a block diagram depicting a configuration example of thedeletion apparatus 932 of FIG. 65.

Referring to FIG. 73, the deletion apparatus 932 includes a degenerationcandidate class selection section 971, a degeneration target classselection section 972, a class degeneration section 973, an evaluationvalue calculation section 974 and a degeneration method selectionsection 975.

To the degeneration candidate class selection section 971, tapcoefficients for each initial class are supplied from the learningapparatus 931 (FIG. 65). Further, to the degeneration candidate classselection section 971, for example, student data (decoding in-progressimage) and teacher data (original image) as learning samples similar tothose supplied to the learning apparatus 931 are supplied.

The degeneration candidate class selection section 971 uses the tapcoefficients for the individual initial classes from the learningapparatus 931 and converts, using the student data (decoding in-progressimage) as a first image, the first image into an equivalent imageequivalent to the teacher data (original image). Further, thedegeneration candidate class selection section 971 uses the equivalentimage and the teacher data to determine a class evaluation value, whichrepresents a performance of a tap coefficient for each initial class,for each initial class, selects degeneration candidate classes, whichbecome candidates as a target of degeneration, from among the initialclasses in response to the class evaluation values, and supplies(information representative of) the degeneration candidate classes tothe degeneration target class selection section 972.

The degeneration candidate class selection section 971 includes an imageconversion section 981 and a selection section 982.

The image conversion section 981 performs, for each initial class,prediction arithmetic operation using tap coefficients of the initialclass and a decoding in-progress image as student data to perform imageconversion for converting the decoding in-progress image as student datainto an equivalent image equivalent to the original image as the teacherdata. Then, the image conversion section 981 supplies the equivalentimage obtained for each initial class to the selection section 982.

The selection section 982 determines a class evaluation value for eachinitial class using the equivalent images for the individual initialclasses from the image conversion section 981 and the original image asthe teacher data. As the class evaluation value, for example, the RDcost, S/N or the like can be adopted.

The selection section 982 uses the class evaluation values of theindividual classes to determine a boundary class as describedhereinabove, for example, with reference to FIG. 63, selects initialclasses whose rank of the class valuation value is lower than that ofthe boundary class as degeneration candidate classes and supplies thedegeneration candidate classes to the degeneration target classselection section 972.

The degeneration target class selection section 972 selects one classthat has not been selected as a degeneration target class as yet or allclasses from among the degeneration candidate classes from thedegeneration candidate class selection section 971 as a degenerationtarget class or classes that become a target of degeneration, andsupplies (information representative of) the degeneration target classor classes to the class degeneration section 973.

It is assumed here that, in order to simplify description, thedegeneration target class selection section 972 selects all of thedegeneration candidate classes from the degeneration candidate classselection section 971 as degeneration target classes.

To the class degeneration section 973, degeneration target glasses aresupplied from the degeneration target class selection section 972 andtap coefficients for each class are supplied from the learning apparatus931 (FIG. 65). Further, to the class degeneration section 973, forexample, student data (decoding in-progress image) and teacher data(original image) as learning samples similar to those supplied to thelearning apparatus 931 as well as encoding information are supplied.

The class degeneration section 973 performs degeneration of classes suchthat degeneration target classes are removed from the initial classes.In particular, for example, for example, the degeneration target classesare integrated into other classes that are not degeneration targetclasses. In this case, pixels to be classified into a degradation targetclass are classified, where a different class after integration is setas a class after degeneration, into the class after degeneration. In theclass degeneration section 973, degeneration of a class is performed byeach of a plurality of kinds of degeneration methods.

Further, the class degeneration section 973 determines (re-determines)tap coefficients for each degeneration class degenerated from initialclasses for each of the plurality of kinds of degeneration methods usingthe student data and teacher data as learning samples and encodinginformation.

Then, the class degeneration section 973 supplies the tap coefficientsfor the individual degeneration classes regarding individual ones of theplurality of kinds of degeneration methods and degeneration informationrepresentative of the degeneration methods to the evaluation valuecalculation section 974 and the degeneration method selection section975.

Here, as the method for degeneration by the class degeneration section973, a tap coefficient reduction method such as, for example, the classcode utilization method, positional relationship utilization method,coefficient selection method, inter-coefficient distance utilizationmethod, seed coefficient utilization method and class evaluation valueutilization method described hereinabove with reference to FIGS. 62 and63 and so forth or any other arbitrary method can be adopted.

To the evaluation value calculation section 974, tap coefficients forindividual ones of the plurality of kinds of degeneration classes (ofindividual ones of the plurality of kinds of degeneration methods) aresupplied from the class degeneration section 973, and further, forexample, student data (decoding in-progress image) and teacher data(original image) as learning samples similar to those supplied to thelearning apparatus 931 as well as encoding information are supplied.

The evaluation value calculation section 974 determines, for each of thetap coefficients for individual ones of the plurality of kinds ofregeneration classes from the class degeneration section 973, adegeneration evaluation value representative of appropriateness of useof tap coefficients for each degeneration class in prediction arithmeticoperation.

The degeneration evaluation value of (the degeneration methods used toobtain) the tap coefficients for each degeneration class is determinedusing, for example, the student data and the teacher data as learningsamples as well as encoding information.

In particular, the evaluation value calculation section 974 includes animage conversion section 991 and a degeneration evaluation valuecalculation section 992.

The image conversion section 991 performs, for each of the tapcoefficients for each of the plurality of kinds of degeneration classesfrom the class degeneration section 973, image conversion for convertinga decoding in-progress image as the student data into an equivalentimage equivalent to the original image as the teacher data by aclassification adaptive process in which the tap coefficients for eachdegeneration class are used, and supplies an equivalent image obtainedby the image conversion to the degeneration evaluation value calculationsection 992.

The degeneration evaluation value calculation section 992 determines,for example, an RD cost, an S/N or the like as a degeneration evaluationvalue for each of the tap coefficients for each of the plurality ofkinds of degeneration classes using the equivalent image obtained foreach of the tap coefficients for each of the plurality of kinds ofregeneration classes and the original image as the teacher data, bothsupplied from the image conversion section 991.

Then, the degeneration evaluation value calculation section 992 suppliesthe degeneration evaluation values for the individual ones of the tapcoefficients for the individual ones of the plurality of kinds ofdegeneration classes to the degeneration method selection section 975.

The degeneration method selection section 975 selects, from among thetap coefficients and the degeneration information for the individualdegeneration classes regarding individual ones of the plurality of kindsof degeneration methods from the class degeneration section 973, the tapcoefficients and the degeneration information for individual ones of thedegeneration classes regarding the degeneration method whosedegeneration value from the evaluation value calculation section 774 isbest as tap coefficients for the individual degeneration classesdegenerated by a degeneration method of a high performance (highperformance degeneration method) and degeneration informationrepresentative of the high performance degeneration method,respectively, and supplies the selected tap coefficients and highperformance degeneration method as reduction filter information to theimage conversion apparatus 933 (FIG. 65) and the reversible encodingsection 106 (FIG. 64).

It is to be noted that, while it is described here that the degenerationtarget class selection section 972 selects all of the degenerationcandidate classes as degeneration target classes and the classdegeneration section 973 performs degeneration of removing all of thedegeneration candidate classes as the degeneration target classes, thedegeneration target class selection section 972 can select the class tobe made a degeneration target class one by one from among thedegeneration target classes.

In this case, the class degeneration section 973 performs degenerationof removing one degeneration target class selected by the degenerationtarget class selection section 972, and a degeneration evaluation valueregarding tap coefficients for each class after the degeneration isdetermined by the evaluation value calculation section 974.

In the case where the degeneration evaluation value regarding tapcoefficients for each class after the degeneration demonstratesimprovement from that in the preceding operation cycle, the degenerationmethod selection section 975 causes the degeneration target classselection section 972 to newly select one class to be made a generationtarget class from among the degeneration target classes as indicated bya broken line arrow mark in FIG. 73 such that similar processes can berepeated thereafter.

On the other hand, in the case where the degeneration evaluation valueregarding tap coefficients for each class after the degeneration doesnot demonstrate improvement from that in the preceding operation cycle,the degeneration method selection section 975 can determine the tapcoefficients for each class after the degeneration obtained at the pointof time of the preceding operation cycle as a final degeneration resultby the degeneration method.

In this case, the class number of classes after degeneration can beoptimized.

Further, while it is described that, in FIG. 73, degenerationinformation is included in the reduction filter information, acorresponding relationship LUT that is a table indicative of acorresponding relationship between initial classes and degenerationclasses may be included in place of the degeneration information in thereduction filter information.

According to the degeneration information, a degeneration class can bedetermined by degenerating initial classes by a degeneration methodrepresented by the degeneration information. Accordingly, while it canbe considered that the corresponding relationship LUT is informationthat directly represents a corresponding relationship between theinitial classes and the degeneration classes, it can be considered thatthe degeneration method is information that indirectly represents thecorresponding relationship between the initial classes and thedegeneration methods.

It is to be noted that the process by the reduction apparatus 932 ofselecting a degeneration method whose degeneration evaluation value isbest and tap coefficients for individual degeneration classesdegenerated by the degeneration method can be performed not only for tapcoefficients for individual initial classes for which classification isperformed by one classification method but also for tap coefficients forindividual ones of a plurality of kinds of initial classes for whichclassification is performed by individual ones of the plurality of kindsof classification methods.

In the case where the process for selecting a degeneration method whosedegeneration evaluation value is best and tap coefficients forindividual ones of degeneration classes degenerated by the degenerationmethod is performed for tap coefficients for individual ones of theplurality of kinds of initial classes for which classification isperformed by individual ones of the plurality of classification methods,a combination of a classification method and a degeneration method whosedegeneration evaluation method such as the RD cost or the like is bestcan be selected from among combinations of individual ones of pluralityof kinds of classification methods and individual ones of the pluralityof kinds of degeneration methods.

<Configuration Example of Image Conversion Section 981>

FIG. 74 is a block diagram depicting a configuration example of theimage conversion section 981 of FIG. 73.

Referring to FIG. 74, the image conversion section 981 includes a tapselection section 711, a coefficient acquisition section 714 and aprediction arithmetic operation section 715.

To the tap selection section 711, a decoding in-progress image asstudent data is supplied.

The tap selection section 711 successively sets pixels of the decodingin-progress image as student data to a noticed pixel, configures aprediction tap similar to that, for example, by the tap selectionsection 941 of FIG. 66 in regard to the noticed pixel and supplies theprediction tap to the prediction arithmetic operation section 715.

To a coefficient acquisition section 714, tap coefficients for eachinitial class from the learning apparatus 931 (FIG. 65) are supplied.

The coefficient acquisition section 714 successively selects the classesas the initial classes as a noticed class, and acquires tap coefficientsof the noticed class from among the tap coefficients for the individualinitial classes from the learning apparatus 931 (FIG. 65) and suppliesthe acquired tap coefficients to the prediction arithmetic operationsection 715.

The prediction arithmetic operation section 715 performs predictionarithmetic operation using the prediction taps from the tap selectionsection 711 and the tap coefficients of the noticed class from thecoefficient acquisition section 714 similarly to the predictionarithmetic operation section 25 of FIG. 31 to generate an equivalentimage equivalent to the original image as the teacher data for eachinitial class, and supplies the equivalent images to the selectionsection 982 (FIG. 73).

<Configuration Example of Image Conversion Section 991>

FIG. 75 is a block diagram depicting a configuration example of theimage conversion section 991 of FIG. 73.

Referring to FIG. 75, the image conversion section 991 includes tapselection sections 721 and 722, a classification section 723, acoefficient acquisition section 724 and a prediction arithmeticoperation section 725.

To the tap selection sections 721 and 722, a decoding in-progress imagedecoding in-progress image as student data is supplied.

The tap selection section 721 successively sets pixels of the decodingin-progress image as the student data as a noticed pixel, and configuresa prediction tap, for example, similar to that by the tap selectionsection 941 of FIG. 66 in regard to the noticed pixel and supplies theprediction tap to the prediction arithmetic operation section 725.

The tap selection section 722 configures a class tap similar to that bythe tap selection section 942 of FIG. 66 in regard to the noticed pixeland supplies the class tap to the classification section 723.

The classification section 723 performs, similarly to the classificationsection 943 of FIG. 66, in regard to the noticed pixel, classificationof the noticed pixel using a plurality of kinds of pixel-relatedinformation, namely, using image characteristic amounts and encodinginformation detected from the class tap from the tap selection section722 and supplies an initial class of the noticed pixel obtained as aresult of the classification to the coefficient acquisition section 724.

To the coefficient acquisition section 724, the initial class of thenoticed pixel is supplied from the classification section 723, and alsodegeneration information from the class degeneration section 973 (FIG.73) and tap coefficients for the individual degeneration classesdegenerated by the degeneration methods represented by the degenerationinformation are supplied.

The coefficient acquisition section 724 converts the initial class ofthe noticed pixel from the classification section 723 into adegeneration class on the basis of the degeneration information from theclass degeneration section 973. Further, the coefficient acquisitionsection 724 acquires tap coefficients of the degeneration class of thenoticed pixel from the tap coefficients for the individual degenerationclasses from the class degeneration section 973 and supplies the tapcoefficients to the prediction arithmetic operation section 725.

The prediction arithmetic operation section 725 performs predictionarithmetic operation using the prediction tap from the tap selectionsection 721 and the tap coefficients of the degeneration class of thenoticed pixel from the coefficient acquisition section 724, for example,similarly to the prediction arithmetic operation section 25 of FIG. 31to finally generate an equivalent image equivalent to the original imageas the teacher data, and supplies the equivalent image to thedegeneration evaluation value calculation section 992 (FIG. 73).

<Configuration Example of Coefficient Acquisition Section 724>

FIG. 76 is a block diagram depicting a configuration example of thecoefficient acquisition section 724 of FIG. 75.

Referring to FIG. 76, the coefficient acquisition section 724 includes astorage section 731, a degeneration class conversion section 732 and anacquisition section 733.

To the storage section 731, a tap coefficient for each degenerationclass is supplied from the class degeneration section 973 (FIG. 73).

The storage section 731 stores tap coefficients for the individualdegeneration classes from the class degeneration section 973.

To the degeneration class conversion section 732, an initial class as aclassification result of a noticed pixel is supplied from theclassification section 723 (FIG. 75). Further, to the degeneration classconversion section 732, degeneration information is supplied from theclass degeneration section 973 (FIG. 73).

The degeneration class conversion section 732 converts the initial classof the notice pixel into a degeneration class of the noticed pixel inaccordance with the degeneration information and supplies thedegeneration class to the acquisition section 733.

The acquisition section 733 acquires tap coefficients of thedegeneration class of the noticed pixel from the degeneration classconversion section 732 from the tap coefficients for the individualdegeneration classes stored in the storage section 731 and supplies theacquired coefficients to the prediction arithmetic operation section 725(FIG. 75).

<Configuration Example of Class Degeneration Section 973>

FIG. 77 is a block diagram depicting a configuration example of theclass degeneration section 973 of FIG. 73.

Referring to FIG. 77, the class degeneration section 973 includes V,which is a plural number, degeneration sections 741 ₁, 741 ₂, . . . ,741 _(v) and V learning sections 742 ₁, 742 ₂, . . . , 742 _(v) andperforms degeneration of tap coefficients for individual initial classesof the learning apparatus 931 by V kinds of degeneration methods.Further, the class degeneration section 973 performs tap coefficientlearning for individual degeneration classes degenerated by individualones of the V degeneration methods to re-determine tap coefficients forthe individual degeneration classes, into which the initial classes aredegenerated, in regard to individual ones of the plurality of kinds ofdegeneration methods.

To a degeneration section 741 _(v), tap coefficients for each initialclass are supplied from the learning apparatus 931 (FIG. 65) anddegeneration target classes are supplied from the degeneration targetclass selection section 972 (FIG. 73).

The degeneration section 741 _(v) performs degeneration of classes bythe vth degeneration method such that the degeneration target classesare removed from the degeneration target class selection section 972from the initial classes whose tap coefficients are supplied from thelearning apparatus 931. Then, the degeneration section 741 _(v) suppliesdegeneration information representative of the vth degeneration methodto a learning section 742 _(v).

To the learning section 742 _(v), not only the degeneration informationis supplied from the degeneration section 741 _(v), but also studentdata (decoding in-progress image) and teacher data (original image) aslearning samples, for example, similar to those supplied to the learningapparatus 931 and encoding information are supplied.

The learning section 742 _(v) uses the student data and the teacher dataas learning samples and the encoding information to perform tapcoefficient learning to determine (re-determine) tap coefficients forthe individual degradation classes degenerated from the initial classes.

In particular, the learning section 742 _(v) converts, in the tapcoefficient learning, the initial class that is a result ofclassification of the noticed pixel into degeneration classes inresponse to the vth degeneration method represented by the degenerationinformation from the degeneration section 741 _(v) and determines tapcoefficients for the individual degeneration classes (hereinafterreferred to also as tap coefficients for the individual degenerationclasses degenerated by the vth degeneration method).

Then, the learning section 742 _(v) supplies the tap coefficients forthe individual degeneration classes degenerated by the vth degenerationmethod and the degeneration information representative of the vthdegeneration method to the evaluation value calculation section 974 andthe degeneration method selection section 975 (FIG. 73).

As the vth degeneration method, one of, for example, as describedhereinabove with reference to FIGS. 62 and 63, the class codeutilization method, positional relationship utilization method,coefficient selection method, inter-coefficient distance utilizationmethod and seed coefficient utilization method described above, tapcoefficient reduction method such as the class evaluation valueutilization method or the like and other arbitrary methods can beadopted solely.

Further, as the vth degeneration method, for example, a combination of aplurality of methods such as a combination of the inter-coefficientdistance utilization method and the seed coefficient utilization method,a combination of the class code utilization method and theinter-coefficient distance utilization method or the like can beadopted.

For example, in the case where the ADRC code is used in classificationto initial classes, a degeneration method of a combination of theinter-coefficient distance utilization method and the class codeutilization method can be adopted. In this case, degeneration of tapcoefficients to each initial class can be performed such that thedegradation evaluation value becomes better and besides the data amountdecreases in comparison with those in an alternative case in which onlythe inter-coefficient distance utilization method or only class codeutilization method is adopted.

Further, for example, in the case where a (significant) physicalquantity that changes continuously such as a DR, DiffMax or the like isused as an image characteristic amount in classification into an initialclass, at least the seed coefficient utilization method that uses one ofor both the DR and DiffMax as a parameter can be adopted as thedegeneration method of tap coefficients for each initial class. Bydegeneration according to the seed coefficient utilization method, thePSNR that has a tradeoff relationship with the compression efficiencycan improved.

Furthermore, for degeneration of tap coefficients for each initialclass, for example, a degeneration method that is a combination of theinter-coefficient distance utilization method and the seed coefficientutilization method can be adopted. Even if a certain initial class c anda different initial class c′ have a small inter-coefficient distance, animage characteristic amount classified into the initial class c and animage characteristic amount classified into the initial class c′sometimes differ significantly from each other. In this case, it issometimes possible to increase the picture quality improvement effect orimprove the compression efficiency (encoding efficiency) by degeneratingtap coefficients of the initial class c and tap coefficients of theinitial class c′ by the seed coefficient utilization method in which theimage characteristic amount is a parameter rather than that bydegenerating the initial classes c and c′ into one class (degenerationclass) by the inter-coefficient distance utilization method.

It is to be noted that, in the case where at least the ADRC code is usedfor classification, especially the class code utilization method isuseful for degeneration of tap coefficients for each initial class.

Further, in the case where at least the ADRC, the DR or DiffMax is usedfor classification, especially the class evaluation value utilizationmethod sometimes is effective for degeneration of tap coefficients foreach initial class.

Furthermore, in the case where the DR or DiffMax is used forclassification, especially the seed-coefficient utilization methodsometimes is effective for degeneration of tap coefficients for eachinitial class.

FIG. 78 is a view illustrating an example of degeneration of tapcoefficients for each initial class by the seed coefficient utilizationmethod.

In FIG. 78, an initial class is configured from an ADRC subclassobtained by subclass classification using the ADRC code and a DRsubclass obtained by subclass classification using the DR.

Further, in FIG. 78, both the ADRC subclass and the DR subclass arerepresented by 2 bits, and the initial class is represented by 4 bitslining up the ADRC subclass of 2 bits and the DR subclass of 2 bits.

Accordingly, the class number of initial classes is 16 (=2⁴) classes.

By degenerating tap coefficients tc # i for individual ones of such 16initial classes as described above by the seed coefficient utilizationmethod in which the DR is made a parameter, a seed coefficient sc # jfor each degeneration class configured rom an ADRC subclass of 2 bits isobtained.

In FIG. 78, for example, tap coefficients tc #0, tc #1, tc #2 and tc #3of four initial classes whose ADRC subclass is 00 and whose DR subclassis 00, 01, 10 and 11, respectively, are degenerated into a seedcoefficient sc #0 of a degeneration class whose ADRC subclass is 00.This similarly applies also to the tap coefficients of the other initialclasses.

From the seed coefficient sc # j for each degeneration class, using theDR as a parameter z, tap coefficients for each degeneration class can begenerated for various values of the DR in accordance with the expression(9).

It is to be noted that, in the case where degeneration of tapcoefficients for each initial class is performed by the seed coefficientutilization method, the degeneration information representative of theseed coefficient utilization method as the degeneration method caninclude information representative of (the kind of) the imagecharacteristic amount used as a parameter. In this case, the decodingapparatus 12 can recognize the parameter from the information includedin the degeneration information and determine tap coefficients from theseed coefficient using the parameter.

FIG. 79 is a block diagram depicting a configuration example of thelearning section 742 _(v) of FIG. 77.

Referring to FIG. 79, the learning section 742 _(v) includes tapselection sections 751 and 752, a classification section 753, anaddition section 754, a coefficient calculation section 755 and adegeneration class conversion section 756.

The tap selection sections 751 and 752 and the classification section753 perform processes similar to those of the tap selection sections 941and 942 and the classification section 752 of the learning apparatus 931of FIG. 66, respectively.

Consequently, to the addition section 754, a prediction tap of a noticedpixel selected from a decoding in-progress result image as student datais supplied from the tap selection section 751, and to the degenerationclass conversion section 756, initial classes are supplied from theclassification section 753.

To the degeneration class conversion section 756, the initial classesare supplied from the classification section 753, and further,degeneration information representative of the vth degeneration methodis supplied from the degeneration section 741 _(v) (FIG. 77).

The degeneration class conversion section 756 converts the initial classof the noticed pixel in accordance with the degeneration method from thedegeneration section 741 _(v) into a degeneration class degenerated bythe vth degeneration method represented by the degeneration information,and supplies the degeneration class to the addition section 754.

The addition section 754 acquires (a pixel value of) a correspondingpixel corresponding to the noticed pixel from among pixels configuringthe original image as teacher data similarly to the addition section 944of FIG. 66, and performs addition of targets of the corresponding pixeland (the pixel value of) the pixels of the decoding in-progress image asthe student data configuring the prediction tap regarding the noticedpixel supplied from the tap selection section 941.

However, the addition section 754 performs the addition not for eachinitial class but for (the class code of) each degeneration classsupplied from the degeneration class conversion section 756.

Then, the addition section 754 sets up a normal equation indicated bythe expression (8) for each degradation class by performing the additionand supplies the normal equations to the coefficient calculation section755.

The coefficient calculation section 755 solves the normal equation foreach degeneration class supplied from the addition section 754 todetermine (re-determine) a tap coefficient (w_(n)) for each degenerationclass degenerated by the vth degeneration method from the number ofinitial classes that can be classified by the classification section 953similarly to the coefficient calculation section 45 of FIG. 13, andsupplies the tap coefficient (w_(n)) to the evaluation value calculationsection 974 and the degeneration method selection section 975 (FIG. 73)together with the degeneration information representative of the vthdegeneration method from the degeneration section 741 _(v) (FIG. 77).

<Configuration Example of Image Conversion Apparatus 933>

FIG. 80 is a block diagram depicting a configuration example of theimage conversion apparatus 933 of FIG. 65.

Referring to FIG. 80, the image conversion apparatus 933 includes tapselection sections 771 and 772, a classification section 773, acoefficient acquisition section 774 and a prediction arithmeticoperation section 775.

The components from the tap selection section 771 to the predictionarithmetic operation section 775 are configured similarly to those fromthe tap selection section 721 to the prediction arithmetic operationsection 725 of FIG. 75, respectively.

To the tap selection sections 771 and 772, a decoding in-progress imageas a first image is supplied.

The tap selection section 771 successively sets pixels of the decodingin-progress image as the first image as a notice pixel and configures aprediction tap, for example, similar to that by the tap selectionsection 941 of FIG. 66 in regard to the noticed pixel, and supplies theconfigured prediction tap to the prediction arithmetic operation section775.

The tap selection section 772 configures a class tap similar to that bythe tap selection section 942 of FIG. 66 in regard to the noticed pixeland supplies the class tap to the classification section 773.

The classification section 773 performs, similarly to the classificationsection 943 of FIG. 66, in regard to a noticed pixel, classificationusing a plurality of kinds of pixel-related information, namely, usingimage characteristic amounts detected from the class tap from the tapselection section 772 and encoding information, and supplies an initialclass of the noticed pixel obtained as a result of the classification tothe coefficient acquisition section 774.

To the coefficient acquisition section 774, not only the initial classof the noticed pixel is supplied from the classification section 773 butalso reduction filter information is supplied from the reductionapparatus 932 (FIG. 65) (FIG. 73).

Here, the reduction filter information supplied from the reductionapparatus 932 to the coefficient acquisition section 774 is reductionfilter information outputted from the degeneration method selectionsection 975 (FIG. 73) of the reduction apparatus 943 and includesdegeneration information representative of a high performancedegeneration method whose degeneration evaluation value is best fromamong the first to vth degeneration methods and tap coefficients foreach degeneration class degenerated from tap coefficients for eachinitial class by the high performance degeneration method.

The coefficient acquisition section 774 converts the initial class ofthe noticed pixel from the classification section 773 into adegeneration class based on the degeneration information included in thereduction filter information from the reduction apparatus 932 by thedegeneration method represented by the degeneration information.Further, the coefficient acquisition section 774 acquires tapcoefficients of the degeneration class of the noticed pixel from amongthe tap coefficients for the individual degeneration classes included inthe reduction filter information from the reduction apparatus 932, andsupplies the tap coefficients to the prediction arithmetic operationsection 775.

The prediction arithmetic operation section 775 performs predictionarithmetic operation using the prediction tap from the tap selectionsection 771 and the tap coefficients of the degeneration class of thenoticed pixel from the coefficient acquisition section 774, for example,similarly to the prediction arithmetic operation section 25 of FIG. 31and finally generates an after-filter image as a second image equivalentto the original image and then supplies the decoding in-progress imageto the frame memory 112 (FIG. 64).

<Configuration Example of Coefficient Acquisition Section 774>

FIG. 81 is a block diagram depicting a configuration example of thecoefficient acquisition section 774 of FIG. 80.

Referring to FIG. 81, the coefficient acquisition section 774 includes astorage section 781, a degeneration class conversion section 782 and anacquisition section 783.

The components from the storage section 781 to the acquisition section783 are configured similarly to those from the storage section 731 tothe acquisition section 733 of FIG. 76, respectively.

To the storage section 781, reduction filter information is suppliedfrom the reduction apparatus 932 (FIG. 65) (FIG. 73).

The storage section 781 stores tap coefficients for the individualdegeneration classes included in the reduction filter information fromthe reduction apparatus 932.

To the degeneration class conversion section 782, an initial class as aclassification result of a noticed pixel is supplied from theacquisition section 773 (FIG. 80). Further, to the degeneration classconversion section 782, reduction filter information is supplied fromthe reduction apparatus 932.

The degeneration class conversion section 782 converts the initial classof the noticed class into a degeneration class of the noticed pixel inaccordance with the degeneration information included in the reductionfilter information and supplies the degeneration class to theacquisition section 783.

The acquisition section 783 acquires tap coefficients of thedegeneration class of the noticed pixel from the degeneration classconversion section 782 from among the tap coefficients for theindividual degeneration classes stored in the storage section 781 andsupplies the acquired tap coefficients to the prediction arithmeticoperation section 775 (FIG. 80).

<Encoding Process>

FIG. 82 is a flow chart illustrating an example of an encoding processof the encoding apparatus 11 of FIG. 64.

At step S311, the learning apparatus 931 (FIG. 65) of the classificationadaptive filter 911 decides whether the timing at present is an updatetiming for a tap coefficient similarly as at step S11 of FIG. 17.

In the case where it is decided at step S311 that the timing at presentis not an update timing for a tap coefficient, the processing advancesto step S320 skipping steps S312 to S319.

On the other hand, in the case where it is decided at step S311 that thetiming at present is an update timing for a tap coefficient, theprocessing advances to step S312, at which the learning apparatus 931performs classification using a plurality of pieces of pixel-relatedinformation to perform tap coefficient learning for determining tapcoefficients for each initial class.

Then, the learning apparatus 931 supplies the tap coefficients for theindividual initial classes obtained by the tap coefficient learning tothe reduction apparatus 932, and the processing advances from step S312to step S313.

At step S313, the degeneration candidate class selection section 971(FIG. 73) of the reduction apparatus 932 selects degeneration candidateclasses from among the initial classes whose tap coefficients aresupplied from the learning apparatus 931 and supplies the selecteddegeneration candidate classes to the degeneration target classselection section 972 (FIG. 73).

The degeneration target class selection section 972 selects adegeneration target class from among the degeneration candidate classesfrom the degeneration candidate class selection section 971 and suppliesthe selected degeneration target class to the class degeneration section973 (FIG. 73), and the processing advances from step S313 to step S314.

At step S314, the class degeneration section 973 degenerates the tapcoefficients for the individual initial classes from the learningapparatus 931 to tap coefficients for individual degeneration classesobtained by removing the degeneration target class (from thedegeneration target class selection section 972) from the initialclasses.

The class degeneration section 973 performs degeneration for the tapcoefficients for the individual initial classes by a plurality of kindsof degeneration methods thereby to determine tap coefficients for eachof the plurality of kinds of degeneration classes.

Then, the class degeneration section 973 supplies, in regard to each ofthe plurality of kinds of degeneration methods, degeneration informationrepresentative of the degeneration method and the tap coefficients forthe individual degeneration classes degenerated by the degenerationmethod to the evaluation value calculation section 974 and thedegeneration method selection section 975 (FIG. 73), and the processingadvances from step S314 to step S315.

At step S315, the evaluation value calculation section 974 determines,in regard to each of the tap coefficients for individual ones of theplurality of kinds of degeneration classes from the class degenerationsection 973, a degeneration evaluation value representative ofappropriateness of use of the tap coefficients for the individualdegeneration classes in prediction arithmetic operation. The evaluationvalue calculation section 974 supplies the degeneration evaluationvalues regarding individual ones of the tap coefficients for individualones of the plurality of kinds of degeneration classes to thedegeneration method selection section 975 (FIG. 73), and the processingadvances from step S315 to step S316.

At step S316, the degeneration method selection section 975 selects,from among the tap coefficients and the degeneration information for theindividual degeneration classes in regard to individual ones of theplurality of kinds of degeneration methods from the class degenerationsection 973, tap coefficients for the individual degeneration classesdegenerated by a high performance degeneration method, and degenerationinformation representative of the high performance degeneration method,namely, the tap coefficients (including seed coefficients as occasiondemands) for the individual degeneration classes regarding thedegeneration method whose degeneration evaluation value is best from theevaluation value calculation section 774 and the degenerationinformation, and acquires them as reduction filter information.

At step S317, the degeneration method selection section 975 of thereduction apparatus 932 outputs the reduction filter information to thereversible encoding section 106 (FIG. 64) and the image conversionapparatus 933 (FIG. 65), and the processing advances to step S318.

At step S318, the reversible encoding section 106 (FIG. 64) sets thereduction filter information from the reduction apparatus 932 as atransmission target, and the processing advances to step S319. Thereduction filter information set as the transmission target is placedinto and transmitted together with encoded data in a prediction encodingprocess performed at step S320 hereinafter described.

At step S319, in the image conversion apparatus 933 (FIG. 80), thestorage section 781 of the coefficient acquisition section 774 (FIG. 81)updates the tap coefficients for the individual degeneration classesstored in the storage section 781 to the tap coefficients for theindividual degeneration classes included in the reduction filterinformation from the reduction apparatus 932 (stores the tapcoefficients for the individual degeneration classes included in thedegeneration filter information in an overwriting state). Then, theprocessing advances to step S320.

At step S320, a prediction encoding process of the original image isperformed, and the encoding process ends.

FIG. 83 is a flow chart illustrating an example of the predictionencoding process at step S320 of FIG. 82.

In the predicting encoding process, at steps S331 to S346, processessimilar to those at steps S31 to S46 of FIG. 18 are performed,respectively.

It is to be noted that, although the classification adaptive filter 911performs, at step S342, a classification adaptive process as a processof an ILF for a decoding in-progress image from the arithmetic operationsection 110 similarly as at step S42 of FIG. 18, in this classificationadaptive process, at step S317 of FIG. 82, tap coefficients forindividual degeneration classes included in reduction filter process tobe outputted from the reduction apparatus 932 to the image conversionapparatus 933 and degeneration information are used.

Further, while, at step S344, the reversible encoding section 106encodes quantization coefficients, encoding information and reductionfilter information similarly as at step S44 of FIG. 18, the reductionfilter information includes tap coefficients for the individualdegeneration classes and the regeneration information.

Accordingly, the encoded data obtained by the reversible encodingsection 106 include the quantization coefficients, encoding information,tap coefficients for the individual degeneration classes as reductionfilter information and degeneration information. Then, such encoded dataare suitably read out from the accumulation buffer 107 and transmittedat step S345 as described hereinabove in connection with step S45 ofFIG. 18.

FIG. 84 is a flow chart illustrating an example of the classificationadaptive process performed at step S342 of FIG. 83.

In the image conversion apparatus 933 (FIG. 80) of the classificationadaptive filter 911, at step S351, the tap selection section 771 selectsone of pixels that have not been selected as a noticed pixel as yet as anoticed pixel from within the decoding in-progress image supplied fromthe arithmetic operation section 110 similarly as at step S51 of FIG.19, and the processing advances to step S352.

At step S352, the tap selection sections 771 and 772 select pixels to bemade a prediction tap and a class tap regarding a noticed pixel fromwithin the decoding in-progress image supplied from the arithmeticoperation section 110 and supply the selected pixels to the predictionarithmetic operation section 775 and the classification section 773,respectively.

Thereafter, the processing advances from step S352 to step S353, atwhich the classification section 773 performs classification of thenoticed pixel using a plurality of kinds of pixel-related informationregarding the noticed pixel such as pixel characteristic amountsdetected from the class tap regarding the noticed pixel, encodinginformation regarding the noticed pixel and so forth.

Then, the classification section 773 supplies an initial class of thenoticed pixel obtained by the classification of the noticed pixel to thecoefficient acquisition section 774, and the processing advances fromstep S353 to step S354.

At step S354, the degeneration class conversion section 782 of thecoefficient acquisition section 774 (FIG. 81) converts the initial classof the noticed pixel supplied from the classification section 773 into adegeneration class of the noticed pixel in accordance with adegeneration method represented by the degeneration information as thereduction filter information supplied from the reduction apparatus 932at step S317 of FIG. 82. Then, the degeneration class conversion section782 supplies the degeneration class of the noticed pixel to theacquisition section 783 (FIG. 81), and the processing advances from stepS354 to step S355.

At step S355, the acquisition section 783 acquires tap coefficients ofthe degeneration class of the noticed pixel from the degeneration classconversion section 782 from among the tap coefficients for theindividual degeneration classes stored in the storage section 781 atstep S319 of FIG. 82 and supplies the acquired tap coefficients to theprediction arithmetic operation section 775, and the processing advancesto step S356.

At steps S356 to S358, processes similar to those at steps S55 to S57 ofFIG. 19 are performed, respectively.

In the encoding apparatus 11 of FIG. 64, classification of a noticedpixel is performed using a plurality of pieces of pixel-relatedinformation in such a manner as described above, and tap coefficientsfor individual ones of a large number of initial classes are determined.Further, the tap coefficients for each initial class are degenerated bya degeneration method selected from among a plurality of kinds ofdegeneration methods, namely, by a degeneration method, for example,whose degeneration evaluation value is best, and tap coefficients forthe individual degeneration classes obtained as a result of thedegeneration and degeneration information representing the degenerationmethod used to obtain the tap coefficients for each degeneration classare transmitted as reduction filter process.

Accordingly, it is possible to improve the compression efficiency andthe S/N of a decoded image.

<Fourth Configuration Example of Decoding Apparatus 12>

FIG. 85 is a block diagram depicting a fourth configuration example ofthe decoding apparatus 12 of FIG. 1.

It is to be noted that, in FIG. 85, elements corresponding to those ofFIG. 20 are denoted by like reference numerals and description of themis hereinafter omitted suitably.

Referring to FIG. 85, the decoding apparatus 12 includes the componentsfrom the accumulation buffer 201 to the arithmetic operation section205, the sorting buffer 207, the D/A conversion section 208, thecomponents from the frame memory 210 to the selection section 214 and aclassification adaptive filter 811.

Accordingly, the decoding apparatus 12 of FIG. 85 is common to that ofFIG. 20 in that it includes the components from the accumulation buffer201 to the arithmetic operation section 205, sorting buffer 207, D/Aconversion section 208 and components from the frame memory 210 to theselection section 214.

However, the decoding apparatus 12 of FIG. 85 is different from that ofFIG. 20 that it includes the classification adaptive filter 811 in placeof the classification adaptive filter 206.

The decoding apparatus 12 of FIG. 85 decodes encoded data transmittedfrom the encoding apparatus 11 of FIG. 64.

Therefore, reduction filter information supplied from the reversibledecoding section 202 to the classification adaptive filter 811 includestap coefficients for individual degeneration classes and degenerationinformation.

The classification adaptive filter 811 is a filter that functions as anILF by performing a classification adaptive process and is common to theclassification adaptive filter 206 of FIG. 20 in that it performs an ILFprocess by a classification adaptive process.

However, the classification adaptive filter 811 is different from theclassification adaptive filter 206 in that it performs a classificationadaptive process using tap coefficients for individual degenerationclasses as reduction filter information and degeneration information.

<Configuration Example of Classification Adaptive Filter 811>

FIG. 86 is a block diagram depicting a configuration example of theclassification adaptive filter 811 of FIG. 85.

Referring to FIG. 86, the classification adaptive filter 811 includes animage conversion apparatus 831.

To the image conversion apparatus 831, a decoding in-progress image issupplied from the arithmetic operation section 205 (FIG. 85) and tapcoefficients and degeneration information for individual degenerationclasses as reduction filter information and encoding information aresupplied from the reversible decoding section 202.

The image conversion apparatus 831 uses, similarly to the imageconversion apparatus 933 of FIG. 65, a decoding in-progress image as thefirst image and performs image conversion by a classification adaptiveprocess using the tap coefficients for the individual degenerationclasses included in the reduction filter information (filtercoefficients obtained using the reduction filter information) and thedegeneration information to convert the decoding in-progress image asthe first image into an after-filter image as a second image equivalentto the original image (to generate an after-filter image), and suppliesthe after-filter image to the sorting buffer 207 and the frame memory210 (FIG. 85).

It is to be noted that the image conversion apparatus 831 performsclassification using the encoding information as occasion demands in aclassification adaptive process similarly to the image conversionapparatus 933 of FIG. 65.

<Configuration Example of Image Conversion Apparatus 831>

FIG. 87 is a block diagram depicting a configuration example of theimage conversion apparatus 831 of FIG. 86.

Referring to FIG. 87, the image conversion apparatus 831 includes tapselection sections 841 and 842, a classification section 843, acoefficient acquisition section 844 and a prediction arithmeticoperation section 845.

To the tap selection sections 841 and 842, a decoding in-progress imageas a first image is supplied from the arithmetic operation section 205(FIG. 85). Further, to the classification section 843, encodinginformation is supplied from the reversible decoding section 202 (FIG.85). Furthermore, to the coefficient acquisition section 844, reductionfilter information is supplied from the reversible decoding section 202.

In the image conversion apparatus 831, the components from the tapselection section 841 to the prediction arithmetic operation section 845perform processing similar to that of the image conversion apparatus 933(FIG. 65) (FIG. 80). Consequently, the decoding in-progress image as thefirst image from the arithmetic operation section 205 (FIG. 85) isconverted into an after-filter image as a second image.

In particular, the tap selection section 841 successively sets pixels ofthe decoding in-progress image as the first image from the arithmeticoperation section 205 as a noticed image and configures, in regard tothe noticed pixel, a prediction tap, for example, similar to that by thetap selection section 771 of FIG. 80, and supplies the prediction tap tothe prediction arithmetic operation section 845.

The tap selection section 842 configures, in regard to the noticedpixel, a class tap similar to that by the tap selection section 772 ofFIG. 80 and supplies the class tap to the classification section 843.

The classification section 843 performs, similarly to the classificationsection 773 of FIG. 80, in regard to the noticed pixel, classificationusing a plurality of kinds of pixel-related information, namely, usingimage characteristic amounts and encoding information detected from theclass tap from the tap selection section 842 and supplies an initialclass of the noticed pixel obtained as a result of the classification tothe coefficient acquisition section 844.

The coefficient acquisition section 844 converts, on the basis of thedegeneration information included in the reduction filter informationfrom the reversible decoding section 202, the initial class of thenotice pixel from the classification section 843 into a degenerationclass by a degeneration method represented by the degenerationinformation. Further, the coefficient acquisition section 844 acquiresthe tap coefficients of the degeneration class of the noticed pixel fromamong the tap coefficients for the individual degeneration classesincluded in the reduction filter information from the reversibledecoding section 202 and supplies the acquired tap coefficients to theprediction arithmetic operation section 845.

The prediction arithmetic operation section 845 performs predictionarithmetic operation using the prediction taps from the tap selectionsection 841 and the tap coefficients of the degeneration class of thenoticed pixel from the coefficient acquisition section 844, for example,similarly to the prediction arithmetic operation section 25 of FIG. 31to finally generate an after-filter image as a second image equivalentto the original image, and supplies the generated after-filter image tothe sorting buffer 207 and the frame memory 210 (FIG. 85).

It is to be noted that, as described hereinabove with reference to FIG.67, in the classification adaptive filter 911 of the encoding apparatus11, a classification method whose RD cost or the like is best isselected as an optimum classification method from among a plurality ofkinds of classification methods, and classification to an initial classcan be performed by the optimum classification method. Further, theencoding apparatus 11 can place and transmit information representativeof the optimum classification method into and together with thereduction filter information to the decoding apparatus 12.

In the decoding apparatus 12, in the case where informationrepresentative of the optimum classification method is included in thereduction filter information, classification to an initial class isperformed by the optimum classification method represented by theinformation.

In particular, in the case where information representative of theoptimum classification method is included in the reduction filterinformation, the classification section 843 configuring the imageconversion apparatus 831 performs classification to an initial class bythe optimum classification method represented by the informationincluded in the reduction filter information.

FIG. 88 is a block diagram depicting a configuration example of thecoefficient acquisition section 844 of FIG. 87.

Referring to FIG. 88, the coefficient acquisition section 844 includes astorage section 851, a degeneration class conversion section 852 and anacquisition section 853. The components from the storage section 851 tothe acquisition section 853 are configured similarly to the componentsfrom the storage section 781 to the acquisition section 783 of FIG. 81,respectively.

To the storage section 851, tap coefficients for individual regenerationclasses included in reduction filter information are supplied from thereversible decoding section 202 (FIG. 85).

The storage section 851 stores tap coefficients for the individualdegeneration classes included in the reduction filter information fromthe reversible decoding section 202.

To the degeneration class conversion section 852, an initial class as aclassification result of a noticed pixel is supplied from theclassification section 843. Further, to the degeneration classconversion section 852, degeneration information included in thereduction filter information is supplied from the reversible decodingsection 202.

The degeneration class conversion section 852 converts the initial classof the noticed pixel into a degeneration class of the noticed pixel inaccordance with the degeneration information included in the reductionfilter information, and supplies the degeneration class to theacquisition section 853.

The acquisition section 853 acquires tap coefficients of thedegeneration class of the noticed pixel from the degeneration classconversion section 852 from among the tap coefficients for theindividual degeneration classes stored in the storage section 851, andsupplies the acquired tap coefficients to the prediction arithmeticoperation section 845.

<Decoding Process>

FIG. 89 is a flow chart illustrating an example of the decoding processof the decoding apparatus 12 of FIG. 85.

In the decoding process, at step S371, the accumulation buffer 201temporarily accumulates encoded data transmitted thereto from theencoding apparatus 11 similarly as at step S71 of FIG. 24 and suitablysupplies the encoded data to the reversible decoding section 202, andthe processing advances to step S372.

At step S372, the reversible decoding section 202 receives and decodesthe encoded data supplied from the accumulation buffer 201 similarly asat step S72 of FIG. 24, and supplies quantization coefficients obtainedby the decoding to the dequantization section 203.

Further, in the case where encoding information or reduction filterinformation is obtained by decoding of the encoded data, the reversibledecoding section 202 supplies necessary encoding information to theintra-prediction section 212, motion prediction compensation section 213and other necessary blocks.

Furthermore, the reversible decoding section 202 supplies the encodinginformation and the reduction filter information to the classificationadaptive filter 811.

Thereafter, the processing advances from step S372 to step S373, atwhich the classification adaptive filter 811 decides whether reductionfilter information is supplied from the reversible decoding section 202.

In the case where it is decided at step S373 that reduction filterinformation is not supplied, the processing advances to step S375skipping step S374.

On the other hand, in the case where it is decided at step S373 thatreduction filter information is supplied, the processing advances tostep S374, at which the coefficient acquisition section 844 configuringthe image conversion apparatus 831 (FIG. 87) of the classificationadaptive filter 811 acquires tap coefficients (including seedcoefficients as occasion demands) for the individual degenerationclasses included in the reduction filter information and thedegeneration information.

Further, in the coefficient acquisition section 844 (FIG. 88), thestorage section 851 updates the tap coefficients for the individualdegeneration classes stored in the storage section 851 to the tapcoefficients for the individual degeneration classes included in thereduction filter information (stores the tap coefficients for theindividual degeneration classes included in the reduction filterinformation in an overwriting state).

Then, the processing advances from step S374 to step S375, at which aprediction decoding process is performed, and the decoding process ends.

FIG. 90 is a flow chart illustrating an example of the predictiondecoding process at step S375 of FIG. 89.

In the prediction decoding process, at steps S381 to S389, processessimilar to those at steps S81 to S89 of FIG. 25 are performed,respectively.

It is to be noted that, although, at step S386, the classificationadaptive filter 811 performs a classification adaptive process as aprocess of an ILF for a decoding in-progress image from the arithmeticoperation section 205 similarly as at step S86 of FIG. 25, in theclassification adaptive process, the tap coefficients for the individualdegeneration classes included in the reduction filter informationacquired by the coefficient acquisition section 844 and the degenerationinformation are used at step S374 of FIG. 89.

FIG. 91 is a flow chart illustrating an example of the classificationadaptive process performed at step S386 of FIG. 90.

In the image conversion apparatus 831 (FIG. 87) of the classificationadaptive filter 811, at step S391, the tap selection section 841 selectsone of pixels that have not been selected as a noticed pixel as yet as anoticed pixel from within the decoding in-progress image supplied fromthe arithmetic operation section 205, and the processing advances tostep S392.

At step S392, the tap selection sections 841 and 842 select pixels to bemade a prediction tap and a class tap regarding a noticed pixel fromwithin the decoding in-progress image supplied from the arithmeticoperation section 205 similarly to the tap selection sections 771 and772 of the image conversion apparatus 933 (FIG. 80) and supply theselected pixels to the classification section 843 and the predictionarithmetic operation section 845, respectively.

Thereafter, the processing advances from step S392 to step S393, atwhich the classification section 843 performs classification of thenoticed pixel using the class tap regarding the noticed pixel and theencoding information regarding the noticed pixel, namely, using theplurality of pieces of pixel-related information of the noticed pixel,similarly to the classification section 773 of the image conversionapparatus 933 (FIG. 80).

Then, the classification section 843 supplies the initial class of thenoticed pixel obtained by the classification of the notice pixel to thecoefficient acquisition section 844, and the processing advances fromstep S393 to step S394.

At step S394, the degeneration class conversion section 852 of thecoefficient acquisition section 844 (FIG. 88) converts the initial classof the noticed pixel supplied from the classification section 843 into adegeneration class of the notice pixel in accordance with thedegeneration information included in the reduction filter informationacquired by the coefficient acquisition section 844 at step S374 of FIG.89. Then, the degeneration class conversion section 852 supplies thedegeneration class of the noticed pixel to the acquisition section 853(FIG. 88), and the processing advances from step S394 to step S395.

At step S395, the acquisition section 853 acquires tap coefficients ofthe degeneration class of the noticed pixel from the degeneration classconversion section 852 from among the tap coefficients for theindividual degeneration classes stored in the storage section 851 atstep S374 of FIG. 89 and supplies the acquired tap coefficients to theprediction arithmetic operation section 845. Then, the processingadvances to step S396.

At steps S396 to S398, processes similar to those at steps S95 to S97 ofFIG. 26 are performed, respectively.

In the encoding apparatus 11 of FIG. 64 and the decoding apparatus 12 ofFIG. 85, since an ILF process is performed by a classification adaptiveprocess as described above, an after-filter image closer to an originalimage than that of a processing result of the ILF can be obtained. As aresult, the S/N of the decoded image can be improved. Further, since anafter-filter image close to the original image can be obtained, theresidual becomes small, and therefore, the compression efficiency can beimproved.

Further, in the encoding apparatus 11, classification to a large numberof initial classes is performed using various pieces of pixel-relatedinformation, and then reduction filter information including tapcoefficients for individual degeneration classes degenerated from tapcoefficients for the individual initial classes by a degeneration methodwhose degeneration evaluation value is best from among a plurality ofkinds of degeneration methods and degeneration informationrepresentative of the degeneration method is transmitted to the decodingapparatus 12. Therefore, the compression efficiency can be improvedfurther.

It is to be noted that, since, in general ALFs, a classification methodor the number of classes obtained by classification is fixed, it hassometimes occurred that a pixel of a decoding in-progress image isclassified into a class that does not appropriately represent a featureof the pixel.

On the other hand, in the classification adaptive filters 911 and 811(FIGS. 64 and 85), since classification of a pixel in a decodingin-progress image is performed using two kinds, three kinds or more ofpixel-related information and classes in the decoding in-progress imageare classified into a large number of classes such as 1000 or more or10000 or more, a pixel can be suppressed from being classified into aclass that does not represent a feature of the pixel appropriately. As aresult, the picture quality of an after-filter image obtained from adecoding in-progress image can be improved.

Furthermore, in the classification adaptive filters 911 and 811, since,using initial classes whose picture quality improvement effect such asAMSE or the like is low from among initial classes obtained byclassification performed using two, three or more kinds of pixel-relatedinformation as degradation target classes, the initial classes aredegenerated to degeneration classes from which (tap coefficients of) thedegeneration target classes are removed, the data amount of the tapcoefficients that become an overhead can be reduced. As a result, whilethe degree of improvement of the picture quality of the after-filterimage is (substantially) maintained, the encoding efficiency(compression efficiency) can be improved.

<Application to Multi-View Image Encoding and Decoding System>

The series of processes described above can be applied to a multi-viewimage encoding and decoding system.

FIG. 92 is a view depicting an example of a multi-view image encodingmethod.

As depicted in FIG. 92, a multi-view image includes images of aplurality of viewpoints (views (view)). A plurality of views of themulti-view image include a base view whose encoding and decoding areperformed using only an image of the own view without using informationof any other view and a non-base view whose encoding and decoding areperformed using information of some other view. Encoding and decoding ofa non-base view may be performed using information of a base view or maybe performed using information of a different non-base view.

In the case where a multi-view image as in the example of FIG. 92 is tobe encoded and decoded, the multi-view image is encoded for eachviewpoint. Then, in the case where encoded data obtained in this mannerare to be decoded, encoded data of each viewpoint are individuallydecoded (namely, for each viewpoint). To such encoding and decoding ofeach viewpoint, the methods described hereinabove in the foregoingdescription of the embodiment may be applied. By such application, theS/N and the compression efficiency can be improved. In short, also inthe case of a multi-view image, the S/N and the compression efficiencycan be improved.

<Multi-View Image Encoding and Decoding System>

FIG. 93 is a view depicting a multi-view image encoding apparatus of amulti-view image encoding and decoding system that performs multi-viewimage encoding and decoding described hereinabove.

As depicted in FIG. 93, the multi-view image encoding apparatus 1000includes an encoding section 1001, another encoding section 1002 and amultiplexing section 1003.

The encoding section 1001 encodes a base view image to generate a baseview image encoded stream. The encoding section 1002 encodes a non-baseview image to generate a non-base view image encoded stream. Themultiplexing section 1003 multiplexes the base view image encoded streamgenerated by the encoding section 1001 and the non-base view imageencoded stream generated by the encoding section 1002 to generate amulti-view image encoded stream.

FIG. 94 is a view depicting a multi-view image decoding apparatus thatperforms multi-view image decoding described hereinabove.

As depicted in FIG. 94, the multi-view image decoding apparatus 1010includes a demultiplexing section 1011, a decoding section 1012 andanother decoding section 1013.

The demultiplexing section 1011 demultiplexes a multi-view image encodedstream in which a base view image encoded stream and a non-base viewimage encoded stream are multiplexed to extract the base view imageencoded stream and the non-base view image encoded stream. The decodingsection 1012 decodes the base view image encoded stream extracted by thedemultiplexing section 1011 to obtain a base view image. The decodingsection 1013 decodes the non-base view image encoded stream extracted bythe demultiplexing section 1011 to obtain a non-base view image.

For example, in such a multi-view image encoding and decoding system asdescribed above, the encoding apparatus 11 described hereinabove in theforegoing description of the embodiment may be applied as the encodingsection 1001 and the encoding section 1002 of the multi-view imageencoding apparatus 1000. By this application, also in encoding of amulti-view image, the methods described in the foregoing description ofthe embodiment can be applied. In other words, it is possible to improvethe S/N and the compression efficiency. Further, for example, as thedecoding section 1012 and the decoding section 1013 of the multi-viewimage decoding apparatus 1010, the decoding apparatus 12 described inthe foregoing description of the embodiment may be applied. By thisapplication, also in decoding of encoded data of a multi-view image, themethods described in the foregoing description of the embodiment can beapplied. In other words, it is possible to improve the S/N and thecompression efficiency.

<Application to Hierarchical Image Encoding and Decoding System>

Further, the series of processes described above can be applied to ahierarchical image encoding (scalable encoding) and decoding system.

FIG. 95 is a view depicting an example of a hierarchical image encodingmethod.

Hierarchical image encoding (scalable encoding) converts image data intoa plurality of layers (hierarchies) of images so as to have ascalability (scalability) function in regard to a predeterminedparameter and encodes the image data for each layer. Hierarchical imagedecoding (scalable decoding) is decoding corresponding to thehierarchical image encoding.

As depicted in FIG. 95, in hierarchization of an image, one image isdivided into a plurality of images (layers) with reference to apredetermined parameter having a scalability function. In short, ahierarchized image (hierarchical image) includes a plurality of imagesof different hierarchies (layers) among which the value of thepredetermined parameter is different. The plurality of layers of thehierarchical image include a base layer that allows encoding anddecoding using only an image of its own layer without utilizing an imageof any other layer and a non-base layer (referred to also as enhancementlayer) that allows encoding and decoding using an image of a differentlayer. The non-base layer may utilize an image of the base layer or mayutilize a different image of the non-base layer.

Generally, a non-base layer is configured from data of a differenceimage (difference data) between an own image of the non-base layer andan image of a different layer such that the redundancy may be reduced.For example, in the case where one image is hierarchized into two layersof a base layer and a non-base layer (also called enhancement layer), animage of lower quality than that of the original image is obtained fromonly data of the base layer, and by synthesizing data of the base layerand data of the non-base layer, the original image (namely, the image ofhigh quality) is obtained.

By hierarchizing an image in this manner, images of various qualitiescan be obtained readily according to the situation. For example, to aterminal having a low processing capacity like a portable telephone set,image compression information only of the base layer (base layer) istransmitted, and a moving image that has a low space time resolution oris not high in picture quality is reproduced. However, to a terminalhaving a high processing capacity like a television set or a personalcomputer, image compression information of the enhancement layer(enhancement layer) is transmitted in addition to the base layer (baselayer), and a moving image that has a high space time resolution or ishigh in picture quality is reproduced. In this manner, image compressioninformation according to the capacity of a terminal or network can betransmitted from a server without performing a transcode process.

In the case where such a hierarchical image as in the example of FIG. 95is encoded and decoded, the hierarchical image is encoded for eachlayer. Then, when the encoded data obtained in this manner are to bedecoded, the encoded data of each other is individually decoded (namely,for each layer). To such encoding and decoding of each layer, themethods described in the above-described embodiment may be applied. Bysuch application, the S/N and the compression efficiency can beimproved. In short, also in the case of a hierarchical image, the S/Nand the compression efficiency can be improved similarly.

<Scalable Parameter>

In such hierarchical image encoding and hierarchical image decoding(scalable encoding and scalable decoding), a parameter having thescalability (scalability) function is arbitrary. For example, a spaceresolution may be made the parameter (spatial scalability). In the caseof this spatial scalability (spatial scalability), the resolution of animage differs for each layer.

Further, as a parameter that has such a scalability property asdescribed above, for example, a time resolution may be applied (temporalscalability). In the case of this temporal scalability (temporalscalability), the frame rate differs for each layer.

Further, as a parameter that has such a scalability property asdescribed above, for example, a signal to noise ratio (SNR (Signal toNoise ratio)) may be applied (SNR scalability). In the case of this SNRscalability (SNR scalability), the SN ratio differs for each layer.

The parameter that has such a scalability property may naturally beother than the examples described above. For example, bit depthscalability (bit-depth scalability) is available in which the base layer(base layer) is configured from an 8-bit (bit) image and a 10-bit (bit)image is obtained by adding an enhancement layer (enhancement layer) tothe 8-bit (bit) image.

Further, chroma scalability (chroma scalability) is available in whichthe base layer (base layer) is configured from a component image of the4:2:0 format and a component image of the 4:2:2 format is obtained byadding an enhancement layer (enhancement layer) to the component image.

<Hierarchical Image Encoding and Decoding System>

FIG. 96 is a view depicting a hierarchical image encoding apparatus of ahierarchical image encoding and decoding system that performs thehierarchical image encoding and decoding described above.

As depicted in FIG. 96, the hierarchical image encoding apparatus 1020includes an encoding section 1021, another encoding section 1022 and amultiplexing section 1023.

The encoding section 1021 encodes a base layer image to generate a baselayer image encoded stream. The encoding section 1022 encodes a non-baseimage to generate a non-base layer image encoded stream. Themultiplexing section 1023 multiplexes the base layer image encodedstream generated by the encoding section 1021 and the non-base layerimage encoded stream generated by the encoding section 1022 to generatea hierarchical image encoded stream.

FIG. 97 is a view depicting a hierarchical image decoding apparatus thatperforms the hierarchical image decoding described hereinabove.

As depicted in FIG. 97, the hierarchical image decoding apparatus 1030includes a demultiplexing section 1031, a decoding section 1032 andanother decoding section 1033.

The demultiplexing section 1031 demultiplexes a hierarchical imageencoded stream in which a base layer image encoded stream and a non-baselayer image encoded stream are multiplexed to extract the base layerimage encoded stream and the non-base layer image encoded stream. Thedecoding section 1032 decodes the base layer image encoded streamextracted by the demultiplexing section 1031 to obtain a base layerimage. The decoding section 1033 decodes the non-base layer imageencoded stream extracted by the demultiplexing section 1031 to obtain anon-base layer image.

For example, in such a hierarchical image encoding and decoding systemas described above, the encoding apparatus 11 described hereinabove inthe foregoing description of the embodiment may be applied as theencoding section 1021 and the encoding section 1022 of the hierarchicalimage encoding apparatus 1020. By this application, also in encoding ofa hierarchical image, the methods described in the foregoing descriptionof the embodiment can be applied. In other words, it is possible toimprove the S/N and the compression efficiency. Further, for example, asthe decoding section 1032 and the decoding section 1033 of thehierarchical image decoding apparatus 1030, the decoding apparatus 12described in the foregoing description of the embodiment may be applied.By this application, also in decoding of encoded data of a hierarchicalimage, the methods described in the foregoing description of theembodiment can be applied. In other words, it is possible to improve theS/N and the compression efficiency.

<Computer>

While the series of processes described above can be executed byhardware, it may otherwise be executed by software. Where the series ofprocesses is executed by software, a program that constructs thesoftware is installed into a computer. Here, the computer includes acomputer incorporated in hardware for exclusive use, a personalcomputer, for example, for universal use that can execute variousfunctions by installing various programs, and so forth.

FIG. 98 is a block diagram depicting an example of a configuration ofhardware of a computer that executes the series of processes describedabove in accordance with a program.

In the computer 1100 depicted in FIG. 98, a CPU (Central ProcessingUnit) 1101, a ROM (Read Only Memory) 1102 and a RAM (Random AccessMemory) 1103 are connected to each other by a bus 1104.

To the bus 1104, also an input/output interface 1110 is connected. Tothe input/output interface 1110, an inputting section 1111, anoutputting section 1112, a storage section 1113, a communication section1114 and a drive 1115 are connected.

The inputting section 1111 is configured, for example, from a keyboard,a mouse, a microphone, a touch panel, an input terminal and so forth.The outputting section 1112 is configured from a display, a speaker, anoutput terminal and so forth. The storage section 1113 is configured,for example, from a hard disk, a RAM disk, a nonvolatile memory and soforth. The communication section 1114 is configured, for example, from anetwork interface. The drive 1115 drives a removable medium 821 such asa magnetic disk, an optical disk, a magneto-optical disk, asemiconductor memory or the like.

In the computer configured in such a manner as described above, the CPU1101 loads a program stored, for example, in the storage section 1113into the RAM 1103 through the input/output interface 1110 and the bus1104 and executes the program to perform the series of processesdescribed above. Into the RAM 1103, also data and so forth necessaryupon execution of various processes by the CPU 1101 are suitably stored.

The program executed by the computer (CPU 1101) can be recorded, forexample, into the removable medium 821 as a package medium or the likeand applied. In this case, the program can be installed into the storagesection 1113 through the input/output interface 1110 by mounting theremovable medium 821 on the drive 1115.

Further, this program can be provided through a wired or wirelesstransmission medium such as a local area network, the Internet or adigital satellite broadcast. In this case, the program can be receivedby the communication section 1114 and installed into the storage section1113.

Also it is possible to install this program in advance into the ROM 1102or the storage section 1113.

<Application of Present Technology>

The encoding apparatus 11 or the decoding apparatus 12 according to theembodiment described hereinabove can be applied to various electronicapparatus such as transmitters or receivers, for example, fordistribution by a satellite broadcast, a wired broadcast such as a cableTV or the Internet, distribution to a terminal by cellular communicationand so forth, or recording apparatus that record an image on a mediumsuch as an optical disk, a magnetic disk, a flash memory or the like,reproduction apparatus for reproducing an image from such storage mediaas described above and so forth. In the following, four examples ofapplication are described.

First Application Example: Television Receiver

FIG. 99 is a view depicting an example of a schematic configuration of atelevision apparatus to which the embodiment described hereinabove isapplied.

The television apparatus 1200 includes an antenna 1201, a tuner 1202, ademultiplexer 1203, a decoder 1204, a video signal processing section1205, a display section 1206, an audio signal processing section 1207, aspeaker 1208, an external interface (I/F) section 1209, a controlsection 1210, a user interface (I/F) section 1211 and a bus 1212.

The tuner 1202 extracts a signal of a desired channel from broadcastingsignals received through the antenna 1201 and demodulates the extractedsignal. Then, the tuner 1202 outputs an encoded bit stream obtained bythe demodulation to the demultiplexer 1203. In particular, the tuner1202 has a role as a transmission section in the television apparatus1200, which receives an encoded stream in which an image is encoded.

The demultiplexer 1203 demultiplexes a video stream and an audio streamof a broadcasting program of a viewing target from an encoded bit streamand outputs the demultiplexed streams to the decoder 1204. Further, thedemultiplexer 1203 extracts auxiliary data such as an EPG (ElectronicProgram Guide) or the like from the encoded bit stream and supplies theextracted data to the control section 1210. It is to be noted that thedemultiplexer 1203 may perform descramble in the case where the encodedbit stream is in a scrambled state.

The decoder 1204 decodes the video stream and the audio stream inputtedfrom the demultiplexer 1203. Then, the decoder 1204 outputs video datagenerated by the decoding process to the video signal processing section1205. Further, the decoder 1204 outputs audio data generated by thedecoding process to the audio signal processing section 1207.

The video signal processing section 1205 reproduces video data inputtedfrom the decoder 1204 and causes the display section 1206 to display avideo. Further, the video signal processing section 1205 may cause thedisplay section 1206 to display an application screen image suppliedthrough the network. Further, the video signal processing section 1205may perform additional processes such as, for example, noise removal andso forth for video data in accordance with a setting. Furthermore, thevideo signal processing section 1205 may generate an image of a GUI(Graphical User Interface) such as, for example, a menu, a button, acursor or the like and cause the generated image to be superimposed onan output image.

The display section 1206 is driven by a drive signal supplied from thevideo signal processing section 1205 and displays a video or an image ona video face of a display device (for example, a liquid crystal display,a plasma display, an OELD (Organic ElectroLuminescence Display) (organicEL display) or the like).

The audio signal processing section 1207 performs a reproduction processsuch as D/A conversion, amplification and so forth for audio datainputted from the decoder 1204 and causes sound to be outputted from thespeaker 1208. Further, the audio signal processing section 1207 mayperform additional processes such as noise removal or the like for theaudio data.

The external interface section 1209 is an interface for connecting thetelevision apparatus 1200 and an external apparatus or a network to eachother. For example, a video stream or an audio stream received throughthe external interface section 1209 may be decoded by the decoder 1204.In particular, also the external interface section 1209 has a role as atransmission section in the television apparatus 1200, which receives anencoded stream in which images are encoded.

The control section 1210 includes a processor such as a CPU, and amemory such as a RAM, a ROM and so forth. The memory stores a program tobe executed by the CPU, program data, EPG data, data acquired through anetwork and so forth. The program stored in the memory is read into andexecuted by the CPU, for example, upon activation of the televisionapparatus 1200. The CPU executes the program to control operation of thetelevision apparatus 1200 in response to an operation signal inputted,for example, from the user interface section 1211.

The user interface section 1211 is connected to the control section1210. The user interface section 1211 includes a button and a switch forallowing, for example, a user to operate the television apparatus 1200,a reception section for a remote controlling signal and so forth. Theuser interface section 1211 detects an operation by the user through thecomponents mentioned and generates an operation signal, and outputs thegenerated operation signal to the control section 1210.

The bus 1212 connects the tuner 1202, demultiplexer 1203, decoder 1204,video signal processing section 1205, audio signal processing section1207, external interface section 1209 and control section 1210 to eachother.

In the television apparatus 1200 configured in such a manner asdescribed above, the decoder 1204 may have a function of the decodingapparatus 12 described hereinabove. In particular, the decoder 1204 maydecode encoded data by a method described hereinabove in connection withthe foregoing embodiment. By such decoding, the television apparatus1200 can improve the S/N and the compression efficiency.

Further, in the television apparatus 1200 configured in such a manner asdescribed above, the video signal processing section 1205 may beconfigured, for example, so as to encode image data supplied from thedecoder 1204 and output the obtained encoded data to the outside of thetelevision apparatus 1200 through the external interface section 1209.Further, the video signal processing section 1205 may have the functionof the encoding apparatus 11 described hereinabove. In short, the videosignal processing section 1205 may encode image data supplied from thedecoder 1204 by the methods described hereinabove in connection with theembodiment. By such encoding, the television apparatus 1200 can improvethe S/N and the compression efficiency.

Second Application Example: Portable Telephone Set

FIG. 100 is a view depicting an example of a schematic configuration ofa portable telephone set to which the embodiment described hereinaboveis applied.

The portable telephone set 1220 includes an antenna 1221, acommunication section 1222, an audio codec 1223, a speaker 1224, amicrophone 1225, a camera section 1226, an image processing section1227, a demultiplexing section 1228, a recording and reproductionsection 1229, a display section 1230, a control section 1231, anoperation section 1232 and a bus 1233.

The antenna 1221 is connected to the communication section 1222. Thespeaker 1224 and the microphone 1225 are connected to the audio codec1223. The operation section 1232 is connected to the control section1231. The bus 1233 connects the communication section 1222, audio codec1223, camera section 1226, image processing section 1227, demultiplexingsection 1228, recording and reproduction section 1229, display section1230 and control section 1231 to each other.

The portable telephone set 1220 performs such operations as transmissionand reception of a voice signal, transmission and reception of anelectronic mail or image data, pickup of an image, recording of data andso forth in various operation modes including a voice speech mode, adata communication mode, an image pickup mode and a videophone mode.

In the voice speech mode, an analog speech signal generated by themicrophone 1225 is supplied to the audio codec 1223. The audio codec1223 converts the analog speech signal into speech data and A/D convertsand compresses the speech data after the conversion. Then, the audiocodec 1223 outputs the speech data after the compression to thecommunication section 1222. Then, the communication section 1222 encodesand modulates the speech data to generate a transmission signal. Then,the communication section 1222 transmits the generated transmissionsignal to a base station (not depicted) through the antenna 1221. On theother hand, the communication section 1222 amplifies and frequencyconverts a wireless signal received through the antenna 1221 to acquirea reception signal. Then, the communication section 1222 demodulates anddecodes the reception signal to generate speech data and outputs thegenerated speech data to the audio codec 1223. The audio codec 1223decompresses and D/A converts the speech data to generate an analogspeech signal. Then, the audio codec 1223 supplies the generated speechsignal to the speaker 1224 such that speech is outputted from thespeaker 1224.

On the other hand, in the data communication mode, for example, thecontrol section 1231 generates character data that configure anelectronic mail in response to operations by the user through theoperation section 1232. Further, the control section 1231 controls thedisplay section 1230 to display characters. Further, the control section1231 generates electronic mail data in response to a transmissioninstruction from the user through the operation section 1232 and outputsthe generated electronic mail data to the communication section 1222.The communication section 1222 encodes and modulates the generatedelectronic mail data to generate a transmission signal. Then, thecommunication section 1222 transmits the generated transmission signalto the base station (not depicted) through the antenna 1221. On theother hand, the communication section 1222 amplifies and frequencyconverts a wireless signal received through the antenna 1221 to acquirea reception signal. Then, the communication section 1222 demodulates anddecodes the reception signal to restore the electronic mail data andoutputs the restored electronic mail data to the control section 1231.The control section 1231 controls the display section 1230 to displaythe substance of the electronic mail and supplies the electronic data tothe recording and reproduction section 1229 such that the electronicdata is written into its recording medium.

The recording and reproduction section 1229 has an arbitrary storagemedium that is readable and writable. For example, the storage mediummay be a built-in type storage medium such as a RAM, a flash memory orthe like or an externally mountable storage medium such as a hard disk,a magnetic disk, a magneto-optical disk, an optical disk, a USB(Universal Serial Bus) memory, a memory card or the like.

Further, in the image pickup mode, for example, the camera section 1226picks up an image of an image pickup object to generate image data andoutputs the generated image data to the image processing section 1227.The image processing section 1227 encodes the image data inputted fromthe camera section 1226 and supplies the encoded stream to the recordingand reproduction section 1229 so as to be written into the storagemedium of the same.

Further, in the image display mode, the recording and reproductionsection 1229 reads out an encoded stream recorded on the storage mediumand outputs the encoded stream to the image processing section 1227. Theimage processing section 1227 decodes the encoded stream inputted fromthe recording and reproduction section 1229 and supplies the image datato the display section 1230 such that the image is displayed.

Further, in the videophone mode, for example, the demultiplexing section1228 multiplexes a video stream encoded by the image processing section1227 and an audio stream inputted from the audio codec 1223 and outputsthe multiplexed stream to the communication section 1222. Thecommunication section 1222 encodes and modulates the stream to generatea transmission signal. Then, the communication section 1222 transmitsthe generated transmission signal to a base station (not depicted)through the antenna 1221. On the other hand, the communication section1222 amplifies and frequency converts a wireless signal received throughthe antenna 1221 to acquire a reception signal. The transmission signaland the reception signal can include an encoded bit stream. Then, thecommunication section 1222 demodulates and decodes the reception signalto restore the stream and outputs the restored stream to thedemultiplexing section 1228. The demultiplexing section 1228demultiplexes the video stream and the audio stream from the inputtedstream and outputs the video stream to the image processing section 1227while it outputs the audio stream to the audio codec 1223. The imageprocessing section 1227 decodes the video stream to generate video data.The video data is supplied to the display section 1230, by which aseries of images are displayed. The audio codec 1223 decompresses andD/A converts the audio stream to generate an analog audio signal. Then,the audio codec 1223 supplies the generated audio signal to the speaker1224 such that speech is outputted from the speaker 1224.

In the portable telephone set 1220 configured in this manner, forexample, the image processing section 1227 may have the function of theencoding apparatus 11 described hereinabove. In short, the imageprocessing section 1227 may encode image data by the methods describedin the foregoing description of the embodiment. By such encoding, theportable telephone set 1220 can improve the S/N and the compressionefficiency.

Further, in the portable telephone set 1220 configured in this manner,for example, the image processing section 1227 may have the function ofthe decoding apparatus 12 described hereinabove. In short, the imageprocessing section 1227 may decode encoded data by the method describedhereinabove in the foregoing description of the embodiment. By suchdecoding, the portable telephone set 1220 can improve the S/N and thecompression efficiency.

Third Application Example: Recording and Reproduction Apparatus

FIG. 101 is a view depicting an example of a schematic configuration ofa recording and reproduction apparatus to which the embodiment describedhereinabove is applied.

The recording and reproduction apparatus 1240 encodes, for example,audio data and video data of a received broadcasting program and recordsthe encoded data on a recording medium. Further, the recording andreproduction apparatus 1240 may encode, for example, audio data andvideo data acquired from a different apparatus and record the data on arecording medium. Further, the recording and reproduction apparatus 1240reproduces, for example, data recorded on the recording medium on amonitor and a speaker in response to an instruction of the user. At thistime, the recording and reproduction apparatus 1240 decodes audio dataand video data.

The recording and reproduction apparatus 1240 includes a tuner 1241, anexternal interface (I/F) section 1242, an encoder 1243, an HDD (HardDisk Drive) section 1244, a disk drive 1245, a selector 1246, a decoder1247, an OSD (On-Screen Display) section 1248, a control section 1249and a user interface (I/F) 1250.

The tuner 1241 extracts a signal of a desired channel from broadcastingsignals received through an antenna (not depicted) and demodulates theextracted signal. Then, the tuner 1241 outputs an encoded bit streamobtained by the demodulation to the selector 1246. In other words, thetuner 1241 has a role as the transmission section in the recording andreproduction apparatus 1240.

The external interface section 1242 is an interface for connecting therecording and reproduction apparatus 1240 and an external apparatus or anetwork to each other. The external interface section 1242 may be, forexample, an IEEE (Institute of Electrical and Electronic Engineers) 1394interface, a network interface, a USB interface, a flash memoryinterface or the like. For example, video data and audio data receivedthrough the external interface section 1242 are inputted to the encoder1243. In other words, the external interface section 1242 has a role asa transmission section in the recording and reproduction apparatus 1240.

The encoder 1243 encodes, in the case where video data and audio datainputted from the external interface section 1242 are not in an encodedstate, the video data and the audio data. Then, the encoder 1243 outputsan encoded bit stream to the selector 1246.

The HDD section 1244 records an encoded bit stream, in which contentdata of videos, audios and so forth are compressed, various programs andother data on an internal hard disk. Further, the HDD section 1244 readsout, upon reproduction of videos and audios, such data from the harddisk.

The disk drive 1245 performs recording and reading out of data on andfrom a recording medium mounted thereon. The recording medium to bemounted on the disk drive 1245 may be, for example, a DVD (DigitalVersatile Disc) disk (DVD-Video, DVD-RAM (DVD-Random Access Memory),DVD-R (DVD-Recordable), DVD-RW (DVD-Rewritable), DVD+R (DVD+Recordable),DVD+RW (DVD+Rewriteable) and so forth) or a Blu-ray (registeredtrademark) disk or the like.

The selector 1246 selects, upon recording of videos and audios, anencoded bit stream inputted from the tuner 1241 or the encoder 1243 andoutputs the selected encoded bit stream to the HDD 1244 or the diskdrive 1245. On the other hand, upon reproduction of videos and audios,the selector 1246 outputs an encoded bit stream inputted from the HDD1244 or the disk drive 1245 to the decoder 1247.

The decoder 1247 decodes an encoded bit stream to generate video dataand audio data. Then, the decoder 1247 outputs the generated video datato the OSD section 1248. Further, the decoder 1247 outputs the generatedaudio data to the external speaker.

The OSD section 1248 reproduces the video data inputted from the decoder1247 and displays a video. Further, the OSD section 1248 may superimposean image of a GUI such as, for example, a menu, a button, a cursor orthe like on the displayed video.

The control section 1249 includes a processor such as a CPU or the likeand a memory such as a RAM, a ROM and so forth. The memory stores aprogram to be executed by the CPU, program data and so forth. Theprogram stored in the memory is read into and executed by the CPU, forexample, upon activation of the recording and reproduction apparatus1240. By executing the program, the CPU controls operation of therecording and reproduction apparatus 1240, for example, in response toan operation signal inputted from the user interface section 1250.

The user interface section 1250 is connected to the control section1249. The user interface section 1250 includes, for example, a buttonand a switch for allowing a user to operate the recording andreproduction apparatus 1240, a reception section for a remotecontrolling signal and so forth. The user interface section 1250 detectsan operation by the user through the components to generate an operationsignal and outputs the generated operation signal to the control section1249.

In the recording and reproduction apparatus 1240 configured in thismanner, for example, the encoder 1243 may have the functions of theencoding apparatus 11 described above. In short, the encoder 1243 mayencode image data by the methods described in the foregoing descriptionof the embodiment. By such encoding, the recording and reproductionapparatus 1240 can improve the S/N and the compression efficiency.

Further, in the recording and reproduction apparatus 1240 configured inthis manner, for example, the decoder 1247 may have the functions of thedecoding apparatus 12 described hereinabove. In short, the decoder 1247may decode encoded data by the methods described in the foregoingdescription of the embodiment. By such decoding, the recording andreproduction apparatus 1240 can improve the S/N and the compressionefficiency.

Fourth Application Example: Image Pickup Apparatus

FIG. 102 is a view depicting an example of a schematic configuration ofan image pickup apparatus to which the embodiment described hereinaboveis applied.

The image pickup apparatus 1260 picks up an image of an image pickupobject to generate an image and encodes and records image data on arecording medium.

The image pickup apparatus 1260 includes an optical block 1261, an imagepickup section 1262, a signal processing section 1263, an imageprocessing section 1264, a display section 1265, an external interface(I/F) section 1266, a memory section 1267, a media drive 1268, an OSDsection 1269, a control section 1270, a user interface (I/F) section1271 and a bus 1272.

The optical block 1261 is connected to the image pickup section 1262.The image pickup section 1262 is connected to the signal processingsection 1263. The display section 1265 is connected to the imageprocessing section 1264. The user interface section 1271 is connected tothe control section 1270. The bus 1272 connects the image processingsection 1264, external interface section 1266, memory section 1267,media drive 1268, OSD section 1269 and control section 1270 to eachother.

The optical block 1261 includes a focus lens, a diaphragm mechanism andso forth. The optical block 1261 forms an optical image of an imagepickup object on an image pick plane of the image pickup section 1262.The image pickup section 1262 includes an image sensor such as a CCD(Charge Coupled Device) image sensor, a CMOS (Complementary Metal OxideSemiconductor) image sensor or the like and converts an optical imageformed on the image pickup plane into an image signal in the form of anelectric signal by photoelectric conversion. Then, the image pickupsection 1262 outputs the image signal to the signal processing section1263.

The signal processing section 1263 performs various camera signalprocesses such as knee correction, gamma correction, color correctionand so forth for an image signal inputted from the image pickup section1262. The signal processing section 1263 outputs the image data afterthe camera signal processes to the image processing section 1264.

The image processing section 1264 encodes the image data inputted fromthe signal processing section 1263 to generate encoded data. Then, theimage processing section 1264 outputs the generated encoded data to theexternal interface section 1266 or the media drive 1268. Further, theimage processing section 1264 decodes encoded data inputted from theexternal interface section 1266 or the media drive 1268 to generateimage data. Then, the image processing section 1264 outputs thegenerated image data to the display section 1265. Further, the imageprocessing section 1264 may output image data inputted from the signalprocessing section 1263 to the display section 1265 such that an imageis displayed. Further, the image processing section 1264 may superimposedisplay data acquired from the OSD section 1269 on the image to beoutputted to the display section 1265.

The OSD section 1269 generates an image of a GUI such as, for example, amenu, a button, a cursor or the like and outputs the generated image tothe image processing section 1264.

The external interface section 1266 is configured, for example, as a USBinput/output terminal. The external interface section 1266 connects theimage pickup apparatus 1260 and a printer, for example, upon printing ofan image. Further, as occasion demands, a drive is connected to theexternal interface section 1266. On the drive, a removable medium suchas, for example, a magnetic disk, an optical disk or the like ismounted, and a program read out from the removable medium can beinstalled into the image pickup apparatus 1260. Furthermore, theexternal interface section 1266 may be configured as a network interfaceconnected to a network such as a LAN, the internet or the like. In otherwords, the external interface section 1266 has a role as a transmissionsection of the image pickup apparatus 1260.

The recording medium to be mounted on the media drive 1268 may be anarbitrary readable and writable removable medium such as, for example, amagnetic disk, a magneto-optical disk, an optical disk, a semiconductormemory or the like. Further, a recording medium may be mounted fixedlyon the media drive 1268 such that a non-portable storage section suchas, for example, a built-in type hard disk drive or an SSD (Solid StateDrive) is configured.

The control section 1270 includes a processor such as a CPU or the likeand a memory such as a RAM, a ROM or the like. The memory stores thereina program to be executed by the CPU, program data and so forth. Theprogram stored in the memory is read into and executed by the CPU, forexample, upon activation of the image pickup apparatus 1260. Byexecuting the program, the CPU controls operation of the image pickupapparatus 1260, for example, in response to an operation signal inputtedfrom the user interface section 1271.

The user interface section 1271 is connected to the control section1270. The user interface section 1271 includes, for example, a button, aswitch and so forth for allowing the user to operate the image pickupapparatus 1260. The user interface section 1271 detects an operation bythe user through the components described to generate an operationsignal and outputs the generated operation signal to the control section1270.

In the image pickup apparatus 1260 configured in this manner, forexample, the image processing section 1264 may have the functions of theencoding apparatus 11 described hereinabove. In short, the imageprocessing section 1264 may encode image data by the method described inthe foregoing description of the embodiment. By such encoding, the imagepickup apparatus 1260 can improve the S/N and the compressionefficiency.

Further, in the image pickup apparatus 1260 configured in such a manneras described above, for example, the image processing section 1264 mayhave the functions of the decoding apparatus 12 described hereinabove.In short, the image processing section 1264 may decode encoded data bythe methods described in the foregoing description of the embodiment. Bysuch decoding, the image pickup apparatus 1260 can improve the S/N andthe compression efficiency.

Other Application Examples

It is to be noted that present technology can be applied to such HTTPstreaming as, for example, MPEG DASH or the like in which appropriatedata is selected and used in a unit of a segment from among a pluralityof encoded data prepared in advance and having resolutions or the likedifferent from each other. In short, also it is possible for such aplurality of encoded data as just described to share informationrelating to encoding or decoding.

Further, while the foregoing description is directed to examples of anapparatus, a system and so forth to which the present technology isapplied, the present technology is not limited to this and can becarried out also as any constitution to be incorporated in such anapparatus as described above or an apparatus that configures such asystem as described above, such as, for example, a processor as a systemLSI (Large Scale Integration) or the like, a module that uses aplurality of processors or the like, a unit that uses a plurality ofmodules or the like, a set in which some other function is added to aunit (namely, some of constitutions of an apparatus).

<Video Set>

An example of a case in which the present technology is carried out as aset is described with reference to FIG. 103.

FIG. 103 is a view depicting an example of a schematic configuration ofa video set to which the present technology is carried out as a set.

In recent years, multifunctionalization of electronic apparatus has beenand is proceeding, and in the case where, in development or manufacture,some configuration is carried out as sales, provision or the like, notonly a case in which it is carried out as a constitution having onefunction, but also a case in which a plurality of constitutions havingfunctions associated with each other are combined and carried out as oneset having a plurality of functions are found increasingly.

The video set 1300 depicted in FIG. 103 has such a multifunctionalizedconfiguration and is a combination, with a device having a function orfunctions relating to encoding and/or decoding of an image (one of orboth encoding and decoding), a device having some other functionrelating to the function or functions.

As depicted in FIG. 103, the video set 1300 includes a module groupincluding a video module 1311, an external memory 1312, a powermanagement module 1313, a front end module 1314 and so forth, and adevice having relating functions such as a connectivity 1321, a camera1322, a sensor 1323 and so forth.

A module is a part in which several part functions related to each otherare collected such that it has coherent functions. Although a particularphysical configuration is arbitrary, for example, a module isconceivable in which electronic circuit elements having individualfunctions such as a plurality of processors, registers, capacitors andso forth and other devices and so forth are disposed on a wiring boardor the like and integrated. Also it is conceivable to combine a modulewith another module, a process or the like to form a new module.

In the case of the example of FIG. 103, the video module 1311 is acombination of constitutions having functions relating to imageprocessing and includes an application processor 1331, a video processor1332, a broadband modem 1333 and an RF module 1334.

A processor includes constitutions, which have predetermined functions,integrated on a semiconductor chip by SoC (System On a Chip) and iscalled, for example, system LSI (Large Scale Integration) or the like.The constitutions having the predetermined functions may be logiccircuits (hardware constitutions), may be a CPU, a ROM, a RAM and soforth and a program executed using them (software configuration) or maybe a combination of both of them. For example, a processor may include alogic circuit and a CPU, a ROM, a RAM and so forth such that some offunctions are implemented by the logic circuit (hardware constitution)and other functions are implemented by a program (softwareconfiguration) executed by the CPU.

The application processor 1331 of FIG. 103 is a processor that executesan application relating to image processing. The application executed bythe application processor 1331 not only can execute, in order toimplement predetermined functions, arithmetic operation processing butalso can control constitutions inside and outside of the video module1311 such as, for example, the video processor 1332 and so forth ifnecessary.

The video processor 1332 is a processor having a function relating toencoding and/or decoding of an image (one of or both encoding anddecoding).

The broadband modem 1333 performs digital modulation or the like fordata (digital signal) to be transmitted by wired or wireless (or bothwired and wireless) broadband communication performed through abroadband line such as the Internet, a public telephone network or thelike to convert the data into an analog signal or converts an analogsignal received by such broadband communication to convert the analogsignal into data (digital signal). The broadband modem 1333 processesarbitrary information such as, for example, image data to be processedby the video processor 1332, a stream encoded from image data, anapplication program, setting data or the like.

The RF module 1334 is a module that performs frequency conversion,modulation/demodulation, amplification, filtering and so forth for an RF(Radio Frequency) signal to be transmitted or received through anantenna. For example, the RF module 1334 performs frequency conversionand so forth for a baseband signal generated by the broadband modem 1333to generate an RF signal. Further, for example, the RF module 1334performs frequency conversion and so forth for an RF signal receivedthrough the front end module 1314 to generate a baseband signal.

It is to be noted that, as indicated by a broken line 1341 in FIG. 103,the application processor 1331 and the video processor 1332 may beintegrated so as to be configured as a single processor.

The external memory 1312 is a module provided outside the video module1311 and having a storage device utilized by the video module 1311.Although the storage device of the external memory 1312 may beimplemented by any physical constitution, since generally it isfrequently utilized for storage of a large amount of data such as imagedata of a unit of a frame, it is desirable to implement the storagedevice by a semiconductor device that is comparatively less expensiveand has a large capacity like a DRAM (Dynamic Random Access Memory).

The power management module 1313 manages and controls power supply tothe video module 1311 (constitutions in the video module 1311).

The front end module 1314 is a module that provides a front end function(circuit at a transmission/reception end of the antenna side) to the RFmodule 1334. As depicted in FIG. 103, the front end module 1314includes, for example, an antenna section 1351, a filter 1352 and anamplification section 1353.

The antenna section 1351 includes an antenna for transmitting andreceiving a wireless signal and peripheral constitutions. The antennasection 1351 transmits a signal supplied from the amplification section1353 as a wireless signal and supplies a received wireless signal as anelectric signal (RF signal) to the filter 1352. The filter 1352 performsfilter processing and so forth for an RF signal received through theantenna section 1351 and supplies the RF signal after the processing tothe RF module 1334. The amplification section 1353 amplifies the RFsignal supplied from the RF module 1334 and supplies the amplified RFsignal to the antenna section 1351.

The connectivity 1321 is a module having functions relating toconnection to the outside. The physical configuration of theconnectivity 1321 is arbitrary. For example, the connectivity 1321includes constitutions having a communication function according to astandard other than a communication standard with which the broadbandmodem 1333 is compatible, external input and output terminals and soforth.

For example, the connectivity 1321 may include a module having acommunication function that complies with a wireless communicationstandard such as Bluetooth (registered trademark), IEEE 802.11 (forexample, Wi-Fi (Wireless fidelity, registered trademark)), NFC (NearField Communication), IrDA (InfraRed Data Association) or the like, anantenna for transmitting and receiving a signal that complies with thestandard, and so forth. Further, for example, the connectivity 1321 mayinclude a module having a communication function that complies with awired communication standard such as USB (Universal Serial Bus), HDMI(registered trademark) (High-Definition Multimedia Interface) or thelike, a terminal that complies with the standard and so forth.Furthermore, for example, the connectivity 1321 may include other data(signal) transmission functions such as analog input and outputterminals and so forth.

It is to be noted that the connectivity 1321 may include a device of atransmission destination of data (signal). For example, the connectivity1321 may include a drive for performing reading out and writing of datafrom and into a recording medium such as a magnetic disk, an opticaldisk, a magneto-optical disk, a semiconductor memory or the like(including not only a drive for a removable medium but also a hard disk,a SSD (Solid State Drive), a NAS (Network Attached Storage) and soforth). Further, the connectivity 1321 may include an outputting deviceof an image or sound (a monitor, a speaker or the like).

The camera 1322 is a module having a function for picking up an image ofan image pickup object to obtain image data of the image pickup object.Image data obtained by image pickup of the camera 1322 is, for example,supplied to and encoded by the video processor 1332.

The sensor 1323 is a module having an arbitrary sensor function such as,for example, a sound sensor, an ultrasonic sensor, an optical sensor, anilluminance sensor, an infrared sensor, an image sensor, a rotationsensor, an angle sensor, an angular velocity sensor, a speed sensor, anacceleration sensor, an inclination sensor, a magnetic identificationsensor, a shock sensor, a temperature sensor and so forth. Data detectedby the sensor 1323 is supplied, for example, to the applicationprocessor 1331 and is utilized by an application or the like.

The constitution described as a module in the foregoing description maybe implemented as a processor, and conversely, the constitutiondescribed as a processor may be implemented as a module.

In the video set 1300 having such a configuration as described above,the present technology can be applied to the video processor 1332 ashereinafter described. Accordingly, the video set 1300 can be carriedout as a set to which the present technology is applied.

<Example of Configuration of Video Processor>

FIG. 104 is a view depicting an example of a schematic configuration ofthe video processor 1332 (FIG. 103) to which the present technology isapplied.

In the case of the example of FIG. 104, the video processor 1332 has afunction for receiving an input of a video signal and an audio signaland encoding them by a predetermined method and a function for decodingencoded video data and audio data and reproducing and outputting a videosignal and an audio signal.

As depicted in FIG. 104, the video processor 1332 includes a video inputprocessing section 1401, a first image scaling section 1402, a secondimage scaling section 1403, a video output processing section 1404, aframe memory 1405 and a memory controlling section 1406. The videoprocessor 1332 further includes an encode-decode engine 1407, video ES(Elementary Stream) buffers 1408A and 1408B and audio ES buffers 1409Aand 1409B. Furthermore, the video processor 1332 includes an audioencoder 1410, an audio decoder 1411, a multiplexing section (MUX(Multiplexer)) 1412, a demultiplexing section (DMUX (Demultiplexer))1413 and a stream buffer 1414.

The video input processing section 1401 acquires a video signalinputted, for example, from the connectivity 1321 (FIG. 103) or the likeand converts the video signal into digital image data. The first imagescaling section 1402 performs format conversion, a scaling process of animage and so forth for image data. The second image scaling section 1403performs a scaling process of an image in response to a format at anoutputting designation through the video output processing section 1404and performs format conversion, a scaling process of an image and soforth similar to those by the first image scaling section 1402 for imagedata. The video output processing section 1404 performs formatconversion, conversion into an analog signal and so forth for image dataand outputs the resulting analog signal as a reproduced video signal,for example, to the connectivity 1321 and so forth.

The frame memory 1405 is a memory for image data shared by the videoinput processing section 1401, first image scaling section 1402, secondimage scaling section 1403, video output processing section 1404 andencode-decode engine 1407. The frame memory 1405 is implemented as asemiconductor memory such as, for example, a DRAM or the like.

The memory controlling section 1406 receives a synchronizing signal fromthe encode-decode engine 1407 and controls writing and reading outaccess to the frame memory 1405 in accordance with an access schedule tothe frame memory 1405 written in an access management table 1406A. Theaccess management table 1406A is updated by the memory controllingsection 1406 in response to a process executed by the encode-decodeengine 1407, first image scaling section 1402, second image scalingsection 1403 or the like.

The encode-decode engine 1407 performs an encoding process of image dataand a decoding process of a video stream that is encoded data of imagedata. For example, the encode-decode engine 1407 encodes image data readout from the frame memory 1405 and successively writes the encoded imagedata as a video stream into the video ES buffer 1408A. Further, forexample, the encode-decode engine 1407 successively reads out anddecodes a video stream from the video ES buffer 1408B and successivelywrites the decoded video stream as image data into the frame memory1405. The encode-decode engine 1407 uses the frame memory 1405 as aworking area in such encoding and decoding. Further, the encode-decodeengine 1407 outputs a synchronizing signal to the memory controllingsection 1406 at a timing at which, for example, processing for eachmacro block is to be started.

The video ES buffer 1408A buffers a video stream generated by theencode-decode engine 1407 and supplies the buffered video stream to themultiplexing section (MUX) 1412. The video ES buffer 1408B buffers avideo stream supplied from the demultiplexing section (DMUX) 1413 andsupplies the buffered video stream to the encode-decode engine 1407.

The audio ES buffer 1409A buffers an audio stream generated by the audioencoder 1410 and supplies the buffered audio stream to the multiplexingsection (MUX) 1412. The audio ES buffer 1409B buffers an audio streamsupplied from the demultiplexing section (DMUX) 1413 and supplies thebuffered audio stream to the audio decoder 1411.

The audio encoder 1410, for example, digitally converts an audio signalinputted from the connectivity 1321 or the like and encodes the digitalaudio signal by a predetermined method such as, for example, an MPEGaudio method, an AC3 (Audio Code number 3) method or the like. The audioencoder 1410 successively writes an audio stream, which is encoded dataof an audio signal, into the audio ES buffer 1409A. The audio decoder1411 decodes an audio stream supplied from the audio ES buffer 1409B,performs, for example, conversion into an analog signal or the like, andsupplies the resulting analog signal as a reproduced audio signal, forexample, to the connectivity 1321 or the like.

The multiplexing section (MUX) 1412 multiplexes a video stream and anaudio stream. The method of the multiplexing (namely, the format of abit stream to be generated by the multiplexing) is arbitrary. Further,upon such multiplexing, also it is possible for the multiplexing section(MUX) 1412 to add predetermined header information and so forth to thebit stream. In other words, the multiplexing section (MUX) 1412 canconvert the format of the stream by the multiplexing. For example, themultiplexing section (MUX) 1412 multiplexes a video stream and an audiostream to convert the streams into a transport stream that is a bitstream of a format for transfer. Further, for example, the multiplexingsection (MUX) 1412 multiplexes a video stream and an audio stream toconvert them into data of a file format for recording (file data).

The demultiplexing section (DMUX) 1413 demultiplexes a bit stream, inwhich a video stream and an audio stream are multiplexed, by a methodcorresponding to that of the multiplexing by the multiplexing section(MUX) 1412. In short, the demultiplexing section (DMUX) 1413 extracts avideo stream and an audio stream from a bit stream read out from thestream buffer 1414 (separates a video stream and an audio stream fromeach other). In short, the demultiplexing section (DMUX) 1413 canconvert the format of a stream by demultiplexing (reverse conversion tothe conversion by the multiplexing section (MUX) 1412). For example, thedemultiplexing section (DMUX) 1413 can convert a transport streamsupplied, for example, from the connectivity 1321, broadband modem 1333or the like into a video stream and an audio stream by acquiring thetransport stream through the stream buffer 1414 and demultiplexing thetransport stream. Further, for example, the demultiplexing section(DMUX) 1413 can convert file data read out from various recording media,for example, by the connectivity 1321 into a video stream and an audiostream by acquiring the file data through the stream buffer 1414 anddemultiplexing the file data.

The stream buffer 1414 buffers a bit stream. For example, the streambuffer 1414 buffers a transport stream supplied from the multiplexingsection (MUX) 1412 and supplies the buffered transport stream, forexample, to the connectivity 1321, broadband modem 1333 and so forth ata predetermined timing or on the basis of a request from the outside orthe like.

Further, the stream buffer 1414 buffers file data supplied from themultiplexing section (MUX) 1412 and supplies the buffered file data, forexample, to the connectivity 1321 and so forth at a predetermined timingor on the basis of a request from the outside or the like such that thefile data is recorded on various recording media.

Furthermore, the stream buffer 1414 buffers a transport stream acquired,for example, through the connectivity 1321, broadband modem 1333 or thelike and supplies the buffered transport stream to the demultiplexingsection (DMUX) 1413 at a predetermined timing or on the basis of arequest from the outside or the like.

Further, the stream buffer 1414 buffers file data read out from variousrecording media, for example, by the connectivity 1321 or the like andsupplies the buffered file data to the demultiplexing section (DMUX)1413 at a predetermined timing or on the basis of a request from theoutside or the like.

Now, an example of operation of the video processor 1332 having such aconfiguration as described above is described. For example, a videosignal inputted from the connectivity 1321 or the like to the videoprocessor 1332 is converted into digital image data of a predeterminedmethod such as a 4:2:2 Y/Cb/Cr method or the like by the video inputprocessing section 1401 and is successively written into the framememory 1405. The digital image data are read out into the first imagescaling section 1402 or the second image scaling section 1403 andsubjected to format conversion to that of a predetermined method such asthe 4:2:0 Y/Cb/Cr method or the like and a scaling process, and then arewritten into the frame memory 1405 again. The image data is encoded bythe encode-decode engine 1407 and written as a video stream into thevideo ES buffer 1408A.

Meanwhile, an audio signal inputted from the connectivity 1321 or thelike to the video processor 1332 is encoded by the audio encoder 1410and written as an audio stream into the audio ES buffer 1409A.

The video stream of the video ES buffer 1408A and the audio stream ofthe audio ES buffer 1409A are read out to and multiplexed by themultiplexing section (MUX) 1412 such that they are converted into atransport stream, file data or the like. The transport stream generatedby the multiplexing section (MUX) 1412 is buffered by the stream buffer1414 and then is outputted to an external network, for example, throughthe connectivity 1321, broadband modem 1333 or the like. Meanwhile, thefile data generated by the multiplexing section (MUX) 1412 is bufferedby the stream buffer 1414 and then outputted, for example, to theconnectivity 1321 or the like and then recorded into various recordingmedia.

On the other hand, a transport stream inputted from an external networkto the video processor 1332, for example, through the connectivity 1321,broadband modem 1333 and so forth is buffered by the stream buffer 1414and then demultiplexed by the demultiplexing section (DMUX) 1413.Meanwhile, file data read out from various recording media, for example,by the connectivity 1321 or the like and inputted to the video processor1332 is buffered by the stream buffer 1414 and then demultiplexed by thedemultiplexing section (DMUX) 1413. In short, a transport stream or filedata inputted to the video processor 1332 is demultiplexed into a videostream and an audio stream by the demultiplexing section (DMUX) 1413.

The audio stream is supplied through the audio ES buffer 1409B to anddecoded by the audio decoder 1411 to reproduce an audio signal.Meanwhile, the video stream is successively read out, after written intothe video ES buffer 1408B, and decoded by the encode-decode engine 1407and written into the frame memory 1405. The decoded image data issubjected to a scaling process by the second image scaling section 1403and is written into the frame memory 1405. Then, the decoded image datais read out into the video output processing section 1404 and formatconverted to a predetermined format such as the 4:2:2 Y/Cb/Cr format,whereafter it is converted into an analog signal and a video signal isreproduced and outputted.

In the case where the present technology is to be applied to the videoprocessor 1332 configured in such a manner as described above, thepresent technology according to the embodiment described above may beapplied to the encode-decode engine 1407. In particular, for example,the encode-decode engine 1407 may have the function of the encodingapparatus 11 or the function of the decoding apparatus 12 describedabove or both of them. This makes it possible for the video processor1332 to obtain advantageous effects similar to those of the encodingapparatus 11 or the decoding apparatus 12 of the embodiment describedhereinabove.

It is to be noted that, in the encode-decode engine 1407, the presenttechnology (namely, the function of the encoding apparatus 11 or thefunction of the decoding apparatus 12 or both of them) may beimplemented by hardware such as logic circuits or the like or may beimplemented by software such as an incorporated program or the like orelse may be implemented by both of them.

<Other Configuration Examples of Video Processor>

FIG. 105 is a view depicting a different example of a schematicconfiguration of the video processor 1332 to which the presenttechnology is applied.

In the case of the example of FIG. 105, the video processor 1332 has afunction for encoding and decoding video data by a predetermined method.

More particularly, as depicted in FIG. 105, the video processor 1332includes a control section 1511, a display interface 1512, a displayengine 1513, an image processing engine 1514 and an internal memory1515. The video processor 1332 further includes a codec engine 1516, amemory interface 1517, a multiplexing and demultiplexing section (MUXDMUX) 1518, a network interface 1519 and a video interface 1520.

The control section 1511 controls operation of the processing sectionsin the video processor 1332 such as the display interface 1512, displayengine 1513, image processing engine 1514, codec engine 1516 and soforth.

As depicted in FIG. 105, the control section 1511 includes, for example,a main CPU 1531, a sub CPU 1532 and a system controller 1533. The mainCPU 1531 executes a program and for forth for controlling operation ofthe processing sections in the video processor 1332. The main CPU 1531generates a control signal in accordance with the program and so forthand supplies the control signal to the processing sections (namely,controls operation of the processing sections). The sub CPU 1532 playsan auxiliary role of the main CPU 1531. For example, the sub CPU 1532executes a child process, a subroutine or the like of the program or thelike executed by the main CPU 1531. The system controller 1533 controlsoperation of the main CPU 1531 and the sub CPU 1532 such as to designateprograms to be executed by the main CPU 1531 and the sub CPU 1532 or thelike.

The display interface 1512 outputs image data, for example, to theconnectivity 1321 and so forth under the control of the control section1511. For example, the display interface 1512 converts image data in theform of digital data into an analog signal and outputs the image data asa reproduced video signal, or the image data of digital data as theyare, to a monitor apparatus or the like of the connectivity 1321.

The display engine 1513 performs various conversion processes such asformat conversion, size conversion, color region conversion and so forthfor the image data under the control of the control section 1511 suchthat the image data satisfies hardware specifications of a monitorapparatus or the like on which an image of the image data is to bedisplayed.

The image processing engine 1514 carries out predetermined imageprocessing such as, for example, a filtering process and so forth forpicture quality improvement for the image data under the control of thecontrol section 1511.

The internal memory 1515 is a memory provided in the inside of the videoprocessor 1332 such that it is shared by the display engine 1513, imageprocessing engine 1514 and codec engine 1516. The internal memory 1515is used for transfer of data performed, for example, between the displayengine 1513, image processing engine 1514 and codec engine 1516. Forexample, the internal memory 1515 stores data supplied from the displayengine 1513, image processing engine 1514 or codec engine 1516 andsupplies, as occasion demands (for example, in accordance with arequest), the data to the display engine 1513, image processing engine1514 or codec engine 1516. Although this internal memory 1515 may beimplemented by any storage device, since generally it is frequentlyutilized for storage of a small amount of data such as image data in aunit of a block, parameters or the like, preferably it is implemented bya semiconductor memory that has a comparatively (for example, incomparison with the external memory 1312) small capacity but is high inresponse speed like, for example, an SRAM (Static Random Access Memory).

The codec engine 1516 performs processing relating to encoding ordecoding of image data. The method for encoding and decoding with whichthe codec engine 1516 is compatible is arbitrary, and the number of suchmethods may be one or a plural number. For example, the codec engine1516 may have codec functions of a plurality of encoding and decodingmethods and perform encoding of image data or decoding of encoded databy a codec function selected from the codec functions.

In the example depicted in FIG. 105, the codec engine 1516 includes, asfunctional blocks for processing relating to the codec, for example, anMPEG-2 Video 1541, an AVC/H.264 1542, an HEVC/H.265 1543, an HEVC/H.265(Scalable) 1544, an HEVC/H.265 (Multi-view) 1545 and an MPEG-DASH 1551.

The MPEG-2 Video 1541 is a functional block that encodes or decodesimage data by the MPEG-2 method. The AVC/H.264 1542 is a functionalblock that encodes or decodes image data by the AVC method. TheHEVC/H.265 1543 is a functional block that encodes or decodes image databy the HEVC method. The HEVC/H.265 (Scalable) 1544 is a functional blockthat scalably encodes or scalably decodes image data by the HEVC method.The HEVC/H.265 (Multi-view) 1545 is a functional block thatmulti-visually encodes or multi-visually decodes image data by the HEVCmethod.

The MPEG-DASH 1551 is a functional block for transmitting and receivingimage data by the MPEG-DASH (MPEG-Dynamic Adaptive Streaming over HTTP)method. MPEG-DASH is a technology for performing streaming of a videousing the HTTP (HyperText Transfer Protocol) and has one ofcharacteristics in that appropriate encoded data is selected andtransmitted in a unit of a segment from among a plurality of encodeddata that are prepared in advance and are different from each other inresolution and so forth. The MPEG-DASH 1551 performs generation of astream that complies with the standard, transmission control of thestream and so forth, and for encoding and decoding of image data, theMPEG-2 Video 1541 to HEVC/H.265 (Multi-view) 1545 described above areutilized.

The memory interface 1517 is an interface for the external memory 1312.Data supplied from the image processing engine 1514 or the codec engine1516 is supplied to the external memory 1312 through the memoryinterface 1517. Meanwhile, data read out from the external memory 1312is supplied to the video processor 1332 (image processing engine 1514 orcodec engine 1516) through the memory interface 1517.

The multiplexing and demultiplexing section (MUX DMUX) 1518 performsmultiplexing and demultiplexing of various data relating to an imagesuch as a bit stream of encoded data, image data, a video signal and soforth. The method for the multiplexing and demultiplexing is arbitrary.For example, upon multiplexing, the multiplexing and demultiplexingsection (MUX DMUX) 1518 not only can collect a plurality of data intoone data but also can add predetermined header information and so forthto the data. Further, upon demultiplexing, the multiplexing anddemultiplexing section (MUX DMUX) 1518 not only can divide one data intoa plurality of data but also can add predetermined header informationand so forth to the divisional data. In short, the multiplexing anddemultiplexing section (MUX DMUX) 1518 can convert the data format bymultiplexing and demultiplexing. For example, the multiplexing anddemultiplexing section (MUX DMUX) 1518 can multiplex bit streams toconvert them into a transport stream that is a bit stream of a formatfor transfer or data of a file format for recording (file data).Naturally, the multiplexing and demultiplexing section (MUX DMUX) 1518can perform inverse conversion by demultiplexing.

The network interface 1519 is an interface, for example, for thebroadband model 1333, connectivity 1321 and so forth. The videointerface 1520 is an interface, for example, for the connectivity 1321,camera 1322 and so forth.

Now, an example of operation of such a video processor 1332 as describedabove is described. For example, if a transport stream is received froman external network through the connectivity 1321, broadband modem 1333or the like, then the transport stream is supplied through the networkinterface 1519 to and demultiplexed by the multiplexing anddemultiplexing section (MUX DMUX) 1518 and then is decoded by the codecengine 1516. Image data obtained by decoding of the codec engine 1516 issubjected, for example, to predetermined image processing by the imageprocessing engine 1514 and further to predetermined conversion by thedisplay engine 1513, and thereafter is supplied, for example, to theconnectivity 1321 or the like through the display interface 1512 suchthat an image thereof is displayed on the monitor. Meanwhile, forexample, image data obtained by decoding of the codec engine 1516 isre-encoded by the codec engine 1516 and multiplexed by the multiplexingand demultiplexing section (MUX DMUX) 1518 such that it is convertedinto file data. The file data is outputted to the connectivity 1321 orthe like through the video interface 1520 and is recorded on variousrecording media.

Furthermore, file data of encoded data, which are encoded image data,read out from a recording medium not depicted, for example, by theconnectivity 1321 or the like are supplied through the video interface1520 to and demultiplexed by the multiplexing and demultiplexing section(MUX DMUX) 1518, and thereafter, they are decoded by the codec engine1516. Image data obtained by the decoding of the codec engine 1516 aresubjected to predetermined image processing by the image processingengine 1514 and further to predetermined conversion by the displayengine 1513, and thereafter, they are supplied through the displayinterface 1512, for example, to the connectivity 1321 or the like suchthat an image thereof is displayed on the monitor. Meanwhile, forexample, image data obtained by decoding of the codec engine 1516 arere-encoded by the codec engine 1516 and multiplexed by the multiplexingand demultiplexing section (MUX DMUX) 1518 such that they are convertedinto a transport stream. The transport stream is supplied, for example,to the connectivity 1321, broadband modem 1333 and so forth through thenetwork interface 1519 and transmitted to a different apparatus notdepicted.

It is to be noted that transfer of image data or other data between theprocessing sections in the video processor 1332 is performed utilizing,for example, the internal memory 1515 or the external memory 1312.Further, the power management module 1313 controls power supply, forexample, to the control section 1511.

In the case where the present technology is applied to the videoprocessor 1332 configured in such a manner as described above, thepresent technology according to the embodiment described hereinabove maybe applied to the codec engine 1516. In short, for example, the codecengine 1516 may have the function of the encoding apparatus 11 or thefunction of the decoding apparatus 12 described above or both of them.By this configuration, the video processor 1332 can achieve advantageouseffects similar to those of the encoding apparatus 11 and the decodingapparatus 12 described hereinabove.

It is to be noted that, in the codec engine 1516, the present technology(namely, the functions of the encoding apparatus 11 and the decodingapparatus 12) may be implemented by hardware such as logic circuits andso forth, may be implemented by software such as an embedded program orthe like or may be implemented by both of them.

While two examples of the configuration of the video processor 1332 areexemplified above, the configuration of the video processor 1332 isarbitrary and may be any other than the two examples described above.Further, although this video processor 1332 may be configured as asingle semiconductor chip, it may otherwise be configured as a pluralityof semiconductor chips. For example, the video processor 1332 may be,for example, a three-dimensional layered LSI in which a plurality ofsemiconductors are stacked. Further, the video processor 1332 may beimplemented by a plurality of LSIs.

Application Example to Apparatus

The video set 1300 can be incorporated into various apparatus thatprocess image data. For example, the video set 1300 can be incorporatedinto the television apparatus 1200 (FIG. 99), portable telephone set1220 (FIG. 100), recording and reproduction apparatus 1240 (FIG. 101),image pickup apparatus 1260 (FIG. 102) and so forth. By incorporatingthe video set 1300 into an apparatus, the apparatus can achieveadvantageous effects similar to those of the encoding apparatus 11 orthe decoding apparatus 12 described hereinabove.

It is to be noted that even some of the constitutions of the video set1300 described above can be carried out as the configuration to whichthe present technology is applied if it includes the video processor1332. For example, only the video processor 1332 can be carried out as avideo processor to which the present technology is applied. Further, forexample, the processor indicated by the broke line 1341, the videomodule 1311 or the like can be carried out as a processor, a module orthe like to which the present technology is applied as describedhereinabove. Furthermore, for example, the video module 1311, externalmemory 1312, power management module 1313 and front end module 1314 canbe combined such that they are carried out as a video unit 1361 to whichthe present technology is applied. In the case of any configuration,advantageous effects similar to those of the encoding apparatus 11 ordecoding apparatus 12 described above can be achieved.

In short, any configuration can be incorporated into various apparatusthat process image data similarly as in the case of the video set 1300if the configuration includes the video processor 1332. For example, thevideo processor 1332, processor indicated by the broken line 1341, videomodule 1311 or video section 1361 can be incorporated into thetelevision apparatus 1200 (FIG. 99), portable telephone set 1220 (FIG.100), recording and reproduction apparatus 1240 (FIG. 101), image pickupapparatus 1260 (FIG. 102) and so forth. Thus, by incorporating any ofthe configurations to which the present technology is applied into anapparatus, the apparatus can achieve advantageous effects similar tothose of the encoding apparatus 11 or the decoding apparatus 12described hereinabove similarly as in the case of the video set 1300.

<Others>

It is to be noted that, although the present specification describes anexample in which various kinds of information are multiplexed intoencoded data (bit stream) and transmitted from the encoding side to thedecoding side, the technique for transmitting such information is notlimited to the example described above. For example, such various kindsof information may be transmitted or recorded as separate dataassociated with encoded data without being multiplexed with the encodeddata. Here, the term “associate” signifies that it is made possible tolink, for example, an image included in the encoded data (such image maybe part of an image such as a slice, a block or the like) andinformation corresponding to the image to each other upon decoding. Inparticular, the information associated with the encoded data (image) maybe transmitted on a transmission line different from that for theencoded data (image). Further, the information associated with theencoded data (image) may be recorded on a recording medium same as thatfor the encoded data (image) (or in a different recording area of a samerecording medium). Furthermore, an image and information thatcorresponds to the image may be associated with each other in anarbitrary unit such as, for example, a plurality of frames, one frame,part in a frame or the like.

Further, the embodiment of the present technology is not limited to theembodiment described hereinabove but can be altered in various mannerswithout departing from the subject matter of the present technology.

For example, in the present specification, the term system signifies aset of plural components (apparatus, modules (parts) and so forth) anddoes not matter whether or not all constitutions are placed in a samehousing. Accordingly, both of a plurality of apparatus that areaccommodated in separate housings and are connected to each other by anetwork and one apparatus in which a plurality of modules areaccommodated in one housing are systems.

Further, for example, a constitution described as one apparatus (or oneprocessing section) may be divided into and configured as a plurality ofapparatus (or processing sections). Conversely, constitutions describedas a plurality of apparatus (or processing sections) in the foregoingdescription may be collected such that they are configured as oneapparatus (or processing section). Further, a constitution other thanthose may naturally be added to the configuration of each apparatus (oreach processing section). Furthermore, if a constitution or operation asan entire system is substantially same, then some of constitutions of acertain apparatus (or a certain processing section) may be included inconstitutions of a different apparatus (or a difference processingsection).

Further, for example, the present technology can assume a configurationfor cloud computing in which one function is shared and processed incooperation by a plurality of apparatus through a network.

Further, for example, the program described hereinabove can be executedby an arbitrary apparatus. In this case, the apparatus may be configuredsuch that it has necessary functions (functional blocks and so forth)and can acquire necessary information.

Further, for example, the steps described in connection with the flowcharts described hereinabove can be executed by one apparatus andfurther can be shared and executed by a plurality of apparatus.Furthermore, in the case where a plurality of processes are included inone step, the plurality of processes included in the one step can beexecuted by one apparatus and also can be shared and executed by aplurality of apparatus.

It is to be noted that the program to be executed by the computer may beof the type by which the processes at steps by which the program isdescribed are executed in a time series in the order as described in thepresent specification or of the type by which the processes are executedin parallel or executed individually at necessary timings such as whenthe process is called. In short, the processes at the steps may beexecuted in an order different from the order described hereinaboveunless inconsistency occurs. Furthermore, the processes at the steps bywhich the program is executed may be executed in parallel to processesof a different program or may be executed in combination with processesof a different apparatus.

It is to be noted that the plurality of present technologies describedin the present specification can individually be carried out solely andindependently of each other unless inconsistency occurs. Naturally, alsoit is possible to carry out an arbitrary plurality of presenttechnologies in combination. For example, also it is possible to carryout the present technology described in the description of anyembodiment in combination with the present technology described in thedescription of a different embodiment. Also it is possible to carry outan arbitrary one of the present technologies described hereinabove incombination with a different technology that is not describedhereinabove.

Further, the advantageous effects described in the present specificationare exemplary to the last and are not restrictive, and otheradvantageous effects may be applicable.

It is to be noted that the present technology can take the followingconfigurations.

<1>

An encoding apparatus, including:

a filter processing section that includes

-   -   a prediction tap selection section configured to select, from        within a first image obtained by addition of a residual of        prediction encoding and a prediction image, pixels that become a        prediction tap to be used for prediction arithmetic operation        for determining a pixel value of a corresponding pixel of a        second image, which corresponds to a processing target pixel        that is a processing target in the first image and is to be used        for prediction of the prediction image,    -   a classification section configured to classify the processing        target pixel to one of a plurality of classes,    -   a tap coefficient acquisition section configured to acquire tap        coefficients of the class of the processing target pixel from        among tap coefficients obtained using reduction filter        information that reduces the tap coefficients for individual        ones of the plurality of classes determined by learning that        uses a student image equivalent to the first image and a teacher        image equivalent to an original image corresponding to the first        image, and    -   an arithmetic operation section configured to determine a pixel        value of the corresponding pixel by performing the prediction        arithmetic operation using the tap coefficients of the class of        the processing target pixel and the prediction tap of the        processing target pixel, the filter processing section        performing a filter process for the first image to generate the        second image; and

a transmission section configured to transmit the reduction filterinformation.

<2>

The encoding apparatus according to <1>, further including:

a reduction section configured to generate the reduction filterinformation.

<3>

The encoding apparatus according to <2>, in which

the reduction section outputs, as the reduction filter information, aselection coefficient that is the latest coefficient of a class selectedfrom among the latest coefficients that are the tap coefficients of theplurality of classes determined by the latest learning.

<4>

The encoding apparatus according to <3>, in which

the reduction section outputs, as the reduction filter information, theselection coefficient of the class selected from among the latestcoefficients of the plurality of classes in response to a merit decisionvalue representative of a degree of a merit in the case where the latestcoefficient is used for the prediction arithmetic operation in place ofa current coefficient that is the tap coefficient at present.

<5>

The encoding apparatus according to <4>, in which

the merit decision value is a value corresponding to

an RD (Rate-Distortion) cost,

an inter-coefficient distance between the latest coefficient and thecurrent coefficient,

an S/N (Signal to Noise Ratio) of the second image determined using thelatest coefficient, or

a use frequency in which the tap coefficient of the class is used forthe prediction arithmetic operation.

<6>

The encoding apparatus according to any one of <3> to <5>, in which

the reduction section outputs, in the case where the latest coefficientof the number of classes equal to or greater than a given number isselected as a selection coefficient from among the latest coefficientsof the plurality of classes, the latest coefficients of all of theplurality of classes are outputted as the reduction filter information.

<7>

The encoding apparatus according to <2>, in which

the reduction section generates

-   -   a tap coefficient for each of integration classes where the        plurality of classes are integrated into the number of classes        equal to or smaller than the plural number of classes, and    -   corresponding relationship information representative of a        corresponding relationship between the plurality of classes and        the integration classes

as the reduction filter information.

<8>

The encoding apparatus according to <7>, in which

the reduction section integrates, in response to a tap coefficientevaluation value representative of appropriateness of use of the tapcoefficient for each of the integration classes in the case where two ormore classes from among the plurality of classes are integrated as anintegration candidate class in the prediction arithmetic operation.

<9>

The encoding apparatus according to <8>, in which

the tap coefficient evaluation value is a value corresponding to

an RD (Rate-Distortion) cost,

an inter-coefficient distance between the tap coefficients of differentclasses,

an S/N (Signal to Noise Ratio) of the second image determined using thetap coefficients,

a use frequency in which the tap coefficient of the class is used in theprediction arithmetic operation, or

a difference between the tap coefficient of a mono class that is aspecific one class and the tap coefficient of a different class.

<10>

The encoding apparatus according to <2>, in which

the reduction section generates a seed coefficient for each of theclasses by which the tap coefficient is determined by given arithmeticoperation with a parameter as the reduction filter information.

<11>

The encoding apparatus according to <10>, further including:

a parameter generation section configured to generate the parameter inresponse to encoding information relating to the prediction encoding ofthe original image, in which

the transmission section transmits parameter information relating to theparameter.

<12>

The encoding apparatus according to <11>, in which

the reduction section generates a seed coefficient of an order selectedin response to a seed coefficient evaluation value representative ofappropriateness of use of the tap coefficient determined from the seedcoefficient in prediction arithmetic operation as the reduction filterinformation.

<13>

The encoding apparatus according to <12>, in which

the seed coefficient evaluation value is a value corresponding to

an RD (Rate-Distortion) cost,

an activity of the original image, or

a code amount target value or a quantization parameter upon predictionencoding of the original image.

<14>

The encoding apparatus according to <2>, in which,

using information relating to the processing target pixel aspixel-related information,

the classification section classifies the processing target pixel to oneof the plurality of classes using a plurality of kinds of thepixel-related information, and

the reduction section generates

-   -   a tap coefficient for each degeneration class after degeneration        of the plurality of classes obtained by degeneration of classes        for reducing the plurality of classes by a degeneration method        selected from among a plurality of kinds of degeneration        methods, and    -   degeneration information representative of the degeneration        method selected from the among the plurality of kinds of        degeneration methods as corresponding relationship information        representative of a corresponding relationship between the        plurality of classes and the degeneration classes

as the reduction filter information.

<15>

The encoding apparatus according to <14>, in which

the reduction section selects the degeneration method in response to adegeneration evaluation value representative of appropriateness of useof individual ones of the tap coefficients for individual ones of aplurality of kinds of the degeneration classes obtained by individualones of the plurality of kinds of degeneration methods in predictionarithmetic operation.

<16>

The encoding apparatus according to <15>, in which

the degeneration evaluation value is a value corresponding to an RD(Rate-Distortion) cost.

<17>

An encoding method, including:

performing a filter process for a first image to generate a secondimage, the performing a filter process including

-   -   selecting, from within the first image that is obtained by        addition of a residual of prediction encoding and a prediction        image, pixels that become a prediction tap to be used for        prediction arithmetic operation for determining a pixel value of        a corresponding pixel of the second image, which corresponds to        a processing target pixel that is a processing target in the        first image and is to be used for prediction of the prediction        image,    -   classifying the processing target pixel to one of a plurality of        classes,    -   acquiring tap coefficients of the class of the processing target        pixel from among tap coefficients obtained using reduction        filter information that reduces the tap coefficients for        individual ones of the plurality of classes determined by        learning that uses a student image equivalent to the first image        and a teacher image equivalent to an original image        corresponding to the first image, and    -   determining a pixel value of the corresponding pixel by        performing the prediction arithmetic operation using the tap        coefficients of the class of the processing target pixel and the        prediction tap of the processing target pixel; and

transmitting the reduction filter information.

<18>

A decoding apparatus, including:

an acceptance section configured to accept reduction filter informationthat reduces tap coefficients for individual ones of a plurality ofclasses determined by learning that uses a student image equivalent to afirst image obtained by adding a residual of prediction encoding and aprediction image and a teacher image equivalent to an original imagecorresponding to the first image; and

a filter processing section that includes

-   -   a prediction tap selection section configured to select, from        within the first image, pixels that become a prediction tap to        be used for prediction arithmetic operation for determining a        pixel value of a corresponding pixel of a second image, which is        used for prediction of the prediction image, corresponding to a        processing target pixel that is a processing target from within        the first image,    -   a classification section configured to classify the processing        target pixel to one of the plurality of classes,    -   a tap coefficient acquisition section configured to acquire a        tap coefficient of the class of the processing target pixel from        the tap coefficients obtained using the reduction filter        information, and    -   an arithmetic operation section configured to determine a pixel        value of the corresponding pixel by performing the prediction        arithmetic operation using the tap coefficient of the class of        the processing target pixel and the prediction tap of the        processing target pixel, the filter processing section        performing a filter process for the first image to generate the        second image.        <19>

The decoding apparatus according to <18>, in which

the reduction filter information is a selection coefficient that is thelatest coefficient of the class selected from among the latestcoefficients that are tap coefficients of the plurality of classesdetermined by the latest learning, and

the tap coefficient acquisition section

stores the tap coefficients for the individual classed, and

updates the tap coefficient of the class of the selection coefficientfrom among the stored tap coefficients for individual ones of theclasses to the selection coefficient.

<20>

The decoding apparatus according to <18>, in which

the reduction filter information is

-   -   a tap coefficient for each integration class when the plurality        of classes are integrated into the number of classes equal to or        smaller than the number of the plurality of classes, and    -   corresponding relationship information representative of a        corresponding relationship between the plurality of classes and        the integration classes, and

the tap coefficient acquisition section

stores the tap coefficients for individual ones of the integrationclasses,

converts the class of the processing target pixel into an integrationclass of the processing target pixel in accordance with thecorresponding relationship information, and

acquires the tap coefficient of the integration class of the processingtarget pixel from the tap coefficients for individual ones of theintegration classes.

<21>

The decoding apparatus according to <18>, in which

the reduction filter information is a seed coefficient for each of theclasses, from which the tap coefficient is determined by givenarithmetic operation with a parameter, and

the tap coefficient acquisition section determines the tap coefficientby the given arithmetic operation between the parameter and the seedcoefficient.

<22>

The decoding apparatus according to <21>, in which

the acceptance section accepts parameter information relating to theparameter generated in response to encoding information relating toprediction encoding of the original image by the encoding side by whichthe prediction encoding of the original image is performed, and

the tap coefficient acquisition section determines the tap coefficientby the given arithmetic operation between a parameter obtained from theparameter information and the seed coefficient.

<23>

The decoding apparatus according to <18>, in which

the reduction filter information is

-   -   a tap coefficient for each of degeneration classes after        degeneration of the plurality of classes obtained by performing        degeneration of classes for reducing the plurality of classes by        a degeneration method selected from among a plurality of kinds        of degeneration methods, and    -   degeneration information, as corresponding relationship        information representative of a corresponding relationship        between the plurality of classes and the degeneration classes,        representative of the degeneration method selected from among        the plurality of kinds of degeneration methods, and

the tap coefficient acquisition section

stores the tap coefficients for individual ones of the degenerationclasses,

converts a class of the processing target pixel into a degenerationclass of the processing target pixel in accordance with the degenerationinformation, and

acquires the tap coefficient of the degeneration class of the processingtarget pixel from among the tap coefficients for individual ones of thedegeneration classes.

<24>

A decoding method, including:

accepting reduction filter information that reduces tap coefficients forindividual ones of a plurality of classes determined by learning thatuses a student image equivalent to a first image obtained by adding aresidual of prediction encoding and a prediction image and a teacherimage equivalent to an original image corresponding to the first image;and

performing a filter process for the first image to generate a secondimage, the performing a filter process including

-   -   selecting, from within the first image, pixels that become a        prediction tap to be used for prediction arithmetic operation        for determining a pixel value of a corresponding pixel of a        second image, which is used for prediction of the prediction        image, corresponding to a processing target pixel that is a        processing target from within the first image,    -   classifying the processing target pixel to one of the plurality        of classes,    -   acquiring a tap coefficient of the class of the processing        target pixel from the tap coefficients obtained using the        reduction filter information, and    -   determining a pixel value of the corresponding pixel by        performing the prediction arithmetic operation using the tap        coefficient of the class of the processing target pixel and the        prediction tap of the processing target pixel.

REFERENCE SIGNS LIST

11 Encoding apparatus, 12 Decoding apparatus, 21, 22 Tap selectionsection, 23 Classification section, 24 Coefficient acquisition section,25 Prediction arithmetic operation section, 30 Learning apparatus, 31Teacher data generation section, 32 Student data generation section, 33Learning section, 41, 42 Tap selection section, 43 Classificationsection, 44 Addition section, 45 Coefficient calculation section, 61Parameter generation section, 62 Student data generation section, 63Learning section, 71 Addition section, 72 Coefficient calculationsection, 81, 82 Addition section, 83 Coefficient calculation section,101 A/D conversion section, 102 Sorting buffer, 103 Arithmetic operationsection, 104 Orthogonal transform section, 105 Quantization section, 106Reversible encoding section, 107 Accumulation buffer, 108 Dequantizationsection, 109 Inverse orthogonal transform section, 110 Arithmeticoperation section, 111 Classification adaptive filter, 112 Frame memory,113 Selection section, 114 Intra-prediction section, 115 Motionprediction compensation section, 116 Prediction image selection section,117 Rate controlling section, 131 Learning apparatus, 132 Reductionapparatus, 133 Image conversion apparatus, 141 Selection section, 151Coefficient acquisition section, 161 Updating section, 162 Storagesection, 163 Acquisition section, 201 Accumulation buffer, 202Reversible decoding section, 203 Dequantization section, 204 Inverseorthogonal transform section, 205 Arithmetic operation section, 206Classification adaptive filter, 207 Sorting buffer, 208 D/A conversionsection, 210 Frame memory, 211 Selection section, 212 Intra-predictionsection, 213 Motion prediction compensation section, 214 Selectionsection, 231 Image conversion apparatus, 241, 242 Tap selection section,243 Classification section, 244 Coefficient acquisition section, 245Prediction arithmetic operation section, 251 Updating section, 252Storage section, 253 Acquisition section, 311 Classification adaptivefilter, 321 Reduction apparatus, 323 Image conversion apparatus, 331Class integration section, 332 Storage section, 333 Correspondingrelationship detection section, 334 Storage section, 341 Coefficientacquisition section, 351 Storage section, 352 Integration classconversion section, 353 Acquisition section, 411 Classification adaptivefilter, 431 Image conversion apparatus, 441 Coefficient acquisitionsection, 451 Storage section, 452 Integration class conversion section,453 Acquisition section, 511 Classification adaptive filter, 531Learning apparatus, 532 Image conversion apparatus, 541 Parametergeneration section, 542 Order setting section, 542 Learning section, 544Selection section, 561 Parameter generation section, 562 Coefficientacquisition section, 571 Storage section, 572 Tap coefficientcalculation section, 573 Storage section, 574 Acquisition section, 611Classification adaptive filter, 631 Image conversion apparatus, 641Coefficient acquisition section, 671 Storage section, 672 Tapcoefficient calculation section, 673 Storage section, 674 Acquisitionsection, 711 Tap selection section, 714 Coefficient acquisition section,715 Prediction arithmetic operation section, 721, 722 Tap selectionsection, 723 Classification section, 724 Coefficient acquisitionsection, 725 Prediction arithmetic operation section, 731 Storagesection, 732 Degeneration class conversion section, 733 Acquisitionsection, 741 _(v) Degeneration section, 742 _(v) Learning section, 751,752 Tap selection section, 753 Classification section, 754 Additionsection, 755 Coefficient calculation section, 756 Degeneration classconversion section, 771, 772 Tap selection section, 773 Classificationsection, 774 Coefficient acquisition section, 775 Prediction arithmeticoperation section, 781 Storage section, 782 Degeneration classconversion section, 783 Acquisition section, 811 Classification adaptivefilter, 831 Image conversion apparatus, 841, 842 Tap selection section,843 Classification section, 844 Coefficient acquisition section, 845Prediction arithmetic operation section, 851 Storage section, 852Degeneration class conversion section, 853 Acquisition section, 911Classification adaptive filter, 931 Learning apparatus, 932 Reductionapparatus, 933 Image conversion apparatus, 940 Learning section, 941,942 Tap selection section, 943 Classification section, 944 Additionsection, 945 Coefficient calculation section, 946 Storage section, 951_(h) Selection section, 952 _(h) Information detection section, 953 _(h)Subclass classification section, 954 Class configuration section, 971Degeneration candidate class selection section, 972 Degeneration targetclass selection section, 973 Class degeneration section, 974 Evaluationvalue calculation section, 975 Degeneration method selection section,981 Image conversion section, 932 Selection section, 991 Imageconversion section, 992 Degeneration evaluation value calculationsection

The invention claimed is:
 1. An encoding apparatus, comprising: a filterprocessing section that includes a prediction tap selection sectionconfigured to select, from within a first image obtained by addition ofa residual of prediction encoding and a prediction image, pixels thatbecome a prediction tap to be used for prediction arithmetic operationfor determining a pixel value of a corresponding pixel of a secondimage, which corresponds to a processing target pixel that is aprocessing target in the first image and is to be used for prediction ofthe prediction image, a classification section configured to classifythe processing target pixel to one of a plurality of classes, a tapcoefficient acquisition section configured to acquire tap coefficientsof the class of the processing target pixel from among tap coefficientsobtained using reduction filter information that reduces the tapcoefficients for individual ones of the plurality of classes determinedby learning that uses a student image equivalent to the first image anda teacher image equivalent to an original image corresponding to thefirst image, and an arithmetic operation section configured to determinea pixel value of the corresponding pixel by performing the predictionarithmetic operation using the tap coefficients of the class of theprocessing target pixel and the prediction tap of the processing targetpixel, the filter processing section performing a filter process for thefirst image to generate the second image; and a transmission sectionconfigured to transmit the reduction filter information, wherein thefilter processing section, the prediction tap selection section, theclassification section, the tap coefficient acquisition section, thearithmetic operation section, and the transmission section are eachimplemented via at least one processor.
 2. The encoding apparatusaccording to claim 1, further comprising: a reduction section configuredto generate the reduction filter information, wherein the reductionsection is implemented via at least one processor.
 3. The encodingapparatus according to claim 2, wherein the reduction section outputs,as the reduction filter information, a selection coefficient that is thelatest coefficient of a class selected from among the latestcoefficients that are the tap coefficients of the plurality of classesdetermined by the latest learning.
 4. The encoding apparatus accordingto claim 3, wherein the reduction section outputs, as the reductionfilter information, the selection coefficient of the class selected fromamong the latest coefficients of the plurality of classes in response toa merit decision value representative of a degree of a merit in a casewhere the latest coefficient is used for the prediction arithmeticoperation in place of a current coefficient that is the tap coefficientat present.
 5. The encoding apparatus according to claim 4, wherein themerit decision value is a value corresponding to an RD (Rate-Distortion)cost, an inter-coefficient distance between the latest coefficient andthe current coefficient, an S/N (Signal to Noise Ratio) of the secondimage determined using the latest coefficient, or a use frequency inwhich the tap coefficient of the class is used for the predictionarithmetic operation.
 6. The encoding apparatus according to claim 3,wherein the reduction section outputs, in a case where the latestcoefficient of the number of classes equal to or greater than a givennumber is selected as a selection coefficient from among the latestcoefficients of the plurality of classes, the latest coefficients of allof the plurality of classes are outputted as the reduction filterinformation.
 7. The encoding apparatus according to claim 2, wherein thereduction section generates a tap coefficient for each of integrationclasses where the plurality of classes are integrated into the number ofclasses equal to or smaller than the plural number of classes, andcorresponding relationship information representative of a correspondingrelationship between the plurality of classes and the integrationclasses as the reduction filter information.
 8. The encoding apparatusaccording to claim 7, wherein the reduction section integrates, inresponse to a tap coefficient evaluation value representative ofappropriateness of use of the tap coefficient for each of theintegration classes in a case where two or more classes from among theplurality of classes are integrated as an integration candidate class inthe prediction arithmetic operation.
 9. The encoding apparatus accordingto claim 8, wherein the tap coefficient evaluation value is a valuecorresponding to an RD (Rate-Distortion) cost, an inter-coefficientdistance between the tap coefficients of different classes, an S/N(Signal to Noise Ratio) of the second image determined using the tapcoefficients, a use frequency in which the tap coefficient of the classis used in the prediction arithmetic operation, or a difference betweenthe tap coefficient of a mono class that is a specific one class and thetap coefficient of a different class.
 10. The encoding apparatusaccording to claim 2, wherein the reduction section generates a seedcoefficient for each of the classes by which the tap coefficient isdetermined by given arithmetic operation with a parameter as thereduction filter information.
 11. The encoding apparatus according toclaim 10, further comprising: a parameter generation section configuredto generate the parameter in response to encoding information relatingto the prediction encoding of the original image, wherein thetransmission section transmits parameter information relating to theparameter, and wherein the parameter generation section is implementedvia at least one processor.
 12. The encoding apparatus according toclaim 10, wherein the reduction section generates a seed coefficient ofan order selected in response to a seed coefficient evaluation valuerepresentative of appropriateness of use of the tap coefficientdetermined from the seed coefficient in prediction arithmetic operationas the reduction filter information.
 13. The encoding apparatusaccording to claim 12, wherein the seed coefficient evaluation value isa value corresponding to an RD (Rate-Distortion) cost, an activity ofthe original image, or a code amount target value or a quantizationparameter upon prediction encoding of the original image.
 14. Theencoding apparatus according to claim 2, wherein, using informationrelating to the processing target pixel as pixel-related information,the classification section classifies the processing target pixel to oneof the plurality of classes using a plurality of kinds of thepixel-related information, and the reduction section generates a tapcoefficient for each degeneration class after degeneration of theplurality of classes obtained by degeneration of classes for reducingthe plurality of classes by a degeneration method selected from among aplurality of kinds of degeneration methods, and degeneration informationrepresentative of the degeneration method selected from the among theplurality of kinds of degeneration methods as corresponding relationshipinformation representative of a corresponding relationship between theplurality of classes and the degeneration classes as the reductionfilter information.
 15. The encoding apparatus according to claim 14,wherein the reduction section selects the degeneration method inresponse to a degeneration evaluation value representative ofappropriateness of use of individual ones of the tap coefficients forindividual ones of a plurality of kinds of the degeneration classesobtained by individual ones of the plurality of kinds of degenerationmethods in prediction arithmetic operation.
 16. The encoding apparatusaccording to claim 15, wherein the degeneration evaluation value is avalue corresponding to an RD (Rate-Distortion) cost.
 17. An encodingmethod, comprising: performing a filter process for a first image togenerate a second image, the performing a filter process includingselecting, from within the first image that is obtained by addition of aresidual of prediction encoding and a prediction image, pixels thatbecome a prediction tap to be used for prediction arithmetic operationfor determining a pixel value of a corresponding pixel of the secondimage, which corresponds to a processing target pixel that is aprocessing target in the first image and is to be used for prediction ofthe prediction image, classifying the processing target pixel to one ofa plurality of classes, acquiring tap coefficients of the class of theprocessing target pixel from among tap coefficients obtained usingreduction filter information that reduces the tap coefficients forindividual ones of the plurality of classes determined by learning thatuses a student image corresponding to the first image and a teacherimage equivalent to an original image corresponding to the first image,and determining a pixel value of the corresponding pixel by performingthe prediction arithmetic operation using the tap coefficients of theclass of the processing target pixel and the prediction tap of theprocessing target pixel; and transmitting the reduction filterinformation.
 18. A decoding apparatus, comprising: an acceptance sectionconfigured to accept reduction filter information that reduces tapcoefficients for individual ones of a plurality of classes determined bylearning that uses a student image equivalent to a first image obtainedby adding a residual of prediction encoding and a prediction image and ateacher image equivalent to an original image corresponding to the firstimage; and a filter processing section that includes a prediction tapselection section configured to select, from within the first image,pixels that become a prediction tap to be used for prediction arithmeticoperation for determining a pixel value of a corresponding pixel of asecond image, which is used for prediction of the prediction image,corresponding to a processing target pixel that is a processing targetfrom within the first image, a classification section configured toclassify the processing target pixel to one of the plurality of classes,a tap coefficient acquisition section configured to acquire a tapcoefficient of the class of the processing target pixel from the tapcoefficients obtained using the reduction filter information, and anarithmetic operation section configured to determine a pixel value ofthe corresponding pixel by performing the prediction arithmeticoperation using the tap coefficient of the class of the processingtarget pixel and the prediction tap of the processing target pixel, thefilter processing section performing a filter process for the firstimage to generate the second image, wherein the acceptance section, thefilter processing section, the prediction tap selection section, theclassification section, the tap coefficient acquisition section, and thearithmetic operation section are each implemented via at least oneprocessor.
 19. The decoding apparatus according to claim 18, wherein thereduction filter information is a selection coefficient that is thelatest coefficient of the class selected from among the latestcoefficients that are tap coefficients of the plurality of classesdetermined by the latest learning, and the tap coefficient acquisitionsection stores the tap coefficients for the individual classed, andupdates the tap coefficient of the class of the selection coefficientfrom among the stored tap coefficients for individual ones of theclasses to the selection coefficient.
 20. The decoding apparatusaccording to claim 18, wherein the reduction filter information is a tapcoefficient for each integration class when the plurality of classes areintegrated into the number of classes equal to or smaller than thenumber of the plurality of classes, and corresponding relationshipinformation representative of a corresponding relationship between theplurality of classes and the integration classes, and the tapcoefficient acquisition section stores the tap coefficients forindividual ones of the integration classes, converts the class of theprocessing target pixel into an integration class of the processingtarget pixel in accordance with the corresponding relationshipinformation, and acquires the tap coefficient of the integration classof the processing target pixel from the tap coefficients for individualones of the integration classes.
 21. The decoding apparatus according toclaim 18, wherein the reduction filter information is a seed coefficientfor each of the classes, from which the tap coefficient is determined bygiven arithmetic operation with a parameter, and the tap coefficientacquisition section determines the tap coefficient by the givenarithmetic operation between the parameter and the seed coefficient. 22.The decoding apparatus according to claim 21, wherein the acceptancesection accepts parameter information relating to the parametergenerated in response to encoding information relating to predictionencoding of the original image by the encoding side by which theprediction encoding of the original image is performed, and the tapcoefficient acquisition section determines the tap coefficient by thegiven arithmetic operation between a parameter obtained from theparameter information and the seed coefficient.
 23. The decodingapparatus according to claim 18, wherein the reduction filterinformation is a tap coefficient for each of degeneration classes afterdegeneration of the plurality of classes obtained by performingdegeneration of classes for reducing the plurality of classes by adegeneration method selected from among a plurality of kinds ofdegeneration methods, and degeneration information, as correspondingrelationship information representative of a corresponding relationshipbetween the plurality of classes and the degeneration classes,representative of the degeneration method selected from among theplurality of kinds of degeneration methods, and the tap coefficientacquisition section stores the tap coefficients for individual ones ofthe degeneration classes, converts a class of the processing targetpixel into a degeneration class of the processing target pixel inaccordance with the degeneration information, and acquires the tapcoefficient of the degeneration class of the processing target pixelfrom among the tap coefficients for individual ones of the degenerationclasses.
 24. A decoding method, comprising: accepting reduction filterinformation that reduces tap coefficients for individual ones of aplurality of classes determined by learning that uses a student imageequivalent to a first image obtained by adding a residual of predictionencoding and a prediction image and a teacher image equivalent to anoriginal image corresponding to the first image; and performing a filterprocess for the first image to generate a second image, the performing afilter process including selecting, from within the first image, pixelsthat become a prediction tap to be used for prediction arithmeticoperation for determining a pixel value of a corresponding pixel of asecond image, which is used for prediction of the prediction image,corresponding to a processing target pixel that is a processing targetfrom within the first image, classifying the processing target pixel toone of the plurality of classes, acquiring a tap coefficient of theclass of the processing target pixel from the tap coefficients obtainedusing the reduction filter information, and determining a pixel value ofthe corresponding pixel by performing the prediction arithmeticoperation using the tap coefficient of the class of the processingtarget pixel and the prediction tap of the processing target pixel.